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Abstract 

Today,  providing  conputer  software  involves  greater  cost  and  risk  than 
providing  computer  equipment.  One  major  reason  is  hardware  is  mass-produced 
by  proven  technology,  while  software  is  still  produced  primarily  by  the  craft 
of  individual  computer  programmers.  The  docunent  is  for  those  who  direct  and 
those  who  implanent  computer  projects;  it  explains  the  selection  and  use  of 
validation,  verification,  and  testing  (V,V&T)  tools  and  techniques  for 
software  development.  A  primary  benefit  of  practicing  V,V&T  is  increasing 
confidence  in  the  quality  of  the  software.  The  docunent  explains  how  to 
develop  a  plan  to  meet  specific  software  V,V&T  goals. 

Key  words:  automated  software  tools;  software  lifecycle;  software  testing; 
software  verification;    test  coverage;    test  data  generation;  validation. 
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CHAPTER  ONE 
INTRODUCTION 

The  Institute  for  Computer  Sciences  and  Technology  (ICST)  carries  out  the 
following  responsibilities  under  P.L.  89-306  (Brooks  Act)  to  improve  the 
Federal  Government's  management  and  use  of  ADP; 

0  develops  Federal  autonatic  data  processing  standards; 

o  provides  agencies  with  scientific  and  technological  advisory  services 
relating  to  ADP; 

o  undertakes  necessary  research  in  computer  sciences  and  technology. 

In  partial  fulfillment  of  Brooks  Act  responsibilities,  ICST  issues  Special 
Publications  (S.P.).  This  docunent  describes  an  approach  to  validation, 
verification,  and  testing  of  computer  software  that  pervades  the  entire 
development  and  maintenance  process.  The  docunent  consists  of  four  additional 
chapters  which  can  be  grouped  into  two  sections,  described  below. 

Software  development  is  an  exercise  in  problem  solving.  The  solution  is 
embodied  in  the  final  product,  the  computer  software.  This  product  consists 
of  the  programs  (computer  instructions)  and  the  manuals  describing  the 
software  and  its  use.  Executing  the  program  with  data  on  a  computer  provides 
the  solution.  During  the  development  of  the  target  software,  there  are 
several  intermediate  products  produced  by  the  project  requestor  and  the 
developers.  The  methodology  to  enhance  the  overall  correctness  of  the  final 
product  by  working  with  the  intermediate  products  is  the  subject  of  this 
document. 

In  problem  solving,  a  key  activity  is  to  determine  that  the  solution  is 
correct.  Validation,  Verification,  and  Testing  (V,V&T)  is  a  process  composed 
of  the  set  of  procedures,  activities,  techniques  and  tools  used  to  ensure  that 
a  software  product  does  solve  the  intended  problem. 

The  chapters  in  this  report  are  grouped  into  two  principal  sections:  V,V&T 
Background  and  V,V&T  Planning  Guide. 

V.V&T  Background  consists  of  two  chapters.  Chapter  2  describes  a  phased 
approach  to  software  development  and  the  fundamental  concepts  of  V,V&T. 
Chapter  3  describes  three  classes  of  V,V&T  techniques  and  tools,  and  a  scheme 
for  integrating  them. 

V.V&T  Planning  Guide  consists  of  two  chapters.  Chapter  4  explains  how  the 
goals  for  V,V&T  can  be  identified  and  hew  to  develop  a  project  V,V&T  plan. 
Chapter  5  discusses  the  application  of  V,V&T  principles  throu^  examples. 
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1.1  AUDIENCE 

Ihis  docunent  is  directed  tcward  those  (managers,  customers,  etc.)  who 
influence  how  software  development  is  done  and  to  those  (programmers, 
analysts,  etc.)  who  do  development.  The  docunent  assists  a  project  manager 
working  with  the  customer  in  establishing  V ,V&T  goals.  It  further  assists 
project  managers  in  developing  a  plan  for  achieving  the  goals.  Finally  it 
guides  in  the  judicious  selection  of  the  appropriate  set  of  V ,V&T  practices, 
techniques,  and  tools. 

To  the  software  engineer  who  is  responsible  for  performing  the  various  V,V&T 
functions,  this  docunent  and  its  companion,  the  Technique  and  Tool  Reference 
Guide[POWE],  provide  guidance  to  the  actual  use  of  the  selected  techniques  and 
tools.  For  each  technique  or  tool  there  is  information  including  functions, 
inputs,  outputs,  resources  required  for  use,  and  sources  for  more  detailed 
information.  In  addition  to  aiding  in  the  application  of  individual 
techniques  and  tools,  information  on  their  integrated  use  is  presented. 

The  document  is  also  a  resource  to  another  group  within  the  ADP  envirorment  in 
that  it  addresses  certain  needs  of  the  ADP  policy  maker.  It  presents  certain 
fundamental  concepts,  elements  of  a  general  V,V&T  approach,  and  many  of  the 
specifics  necessary  to  implement  the  approach.  It  provides  information  that 
may  be  helpful  in  forming  a  basis  for  policies  relating  to  V,V&T  practice. 
Second,  the  docunent  contains  information  relevant  to  the  formulation  and 
inplanentation  of  the  software  V,V&T  functions  in  a  typical  ADP  envirorment. 

1.2  PHILOSOPHY 

This  docunent  provides  assistance  in  all  aspects  of  V,V&T.  It  is  not  a  total 
"project  cookbod<,"  wherein  all  the  techniques  and  tools  must  be  used  on  every 
project.  Projects  vary  in  size,  scope,  ccmplexity,  and  other  characteristics 
that  influence  the  specific  V,V&T  approach.  Each  project  must  be  evaluated  to 
determine  how  V,V&T  might  be  applied. 

The  use  of  V,V&T  does  not,  of  itself,  guarantee  success.  It  requires 
judgement,  training,  and  experience.  Like  other  methodologies,  the 
application  of  V,V&T  may  be  good  or  bad;    but,  it  should  never  be  blind. 

1.3  USER  OF  IHIS  DOCUMENT 

For  the  reader  who  is  encountering  this  docunent  for  the  first  time,  reading 
Chapters  1  through  3  is  reconmended.  Reading  of  subsequent  chapters  should  be 
guided  by  intended  use  of  the  information.  Suggested  reading  scenarios  for 
three  types  of  readers  are  presented  in  Figure  1.1-1.  "Validation, 
Verification,  and  Testing  -  Technique  and  Tool  Guide"[POWE]  is  a  companion  to 
the  general  guidance  in  this  docunent. 


V,V&T  BACKGROUND  V,V&T  PLANNING  GUIDE 

Chapters  12         3  4  5 

-  POLICY  MAKER  -  _    -    -  PLANNER  -    -  - 

-    -    -  SELECTOR  -    -  - 


Figure  1.1-1    Reading  Scenarios 
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CHAPTER  IWO 

AN  OVERVIEW  OF  SOFTVARE  DEVELOPMENT 

Systematic  approaches  to   problem   solving,    including    software  development, 
proceed    through    a    sequence  of  steps  or  phases  with  probable  looping  back  to 
previous  phases.    This  type  of  approach  yields  several  important  benefits. 

First,  because  the  problem  solving  process  is  subdivided  into  separate  phases, 
the  problem  solver  is  eased  into  a  gradual  process.  The  problem  solver  is 
able  to  approach  separately  the  problem,  the  solution,  and  its  implementation. 
The  problem  solver  need  not  deliberate  on  all  three  at  once.  This  allows 
larger,  more  complex  problems  to  be  resolved  successfully. 

Second,  a  phased  process  provides  intermediate  monitoring  and  control  points. 
The  existence  of  phases  and  intermediate  products  increases  the  visibility  of 
the  process.  This  encourages  the  involvement  of  the  group  for  whom  the 
problem  is  being  solved  and  increases  their  control  of  the  problem  solving. 

Third,  the  existence  of  a  series  of  intermediate  specifications,  i.e. 
requironents,  design,  and  code,  of  the  solution  facilitates  early  and 
continual  evaluation  of  the  solution  as  it  moves  toward  implementation.  In 
software  development  this  process  of  continual  review  and  evaluation  is 
validation,  verification,  and  testing  (V,V&T). 

This  chapter  discusses  software  development  as  a  problem  solving  activity.  It 
introduces  the  concepts  of  software  V ,V&T  and  a  phased  approach  to  software 
development.    Specifically,  it  describes: 

Software  Validation,  Verification,  and  Testing  -  Evaluation  and  Review 
Throughout  the  Problem  Solving  Process  -  The  process  of  obtaining  increasing 
levels  of  confidence  in  a  solution  through  a  series  of  checkpoints  and  reviews 
is  discussed  as  it  applies  to  software  development  and  the  analogies 
previously  presented. 

A  Problem  Solving  Model  for  Software  Develocment  -  A  specific  series  of 
problem  solving  phases  for  software  development  is  described,  including  the 
V,V&T  activities  associated  with  each. 

2.1  SOFTWARE  VALIDATION,  VERIFICATION,  AND  TESTING  -  EVALUATION  AND  REVIEW 
THROUGHOUT  THE  PROBLEM  SOLVING  PROCESS 

Validation,  Verification,  and  Testing  (V,V&T),  in  general  terms,  is  a  process 
of  review,  analysis,  and  testing  employed  throu^out  the  software  development 
lifecycle.  It  is  a  methodology  which  helps  ensure  the  production  of  quality 
software.  Validation  determines  the  correctness  of  the  end  product,  e.g. 
code,  with  respect  to  the  software  requironents,  i.e.  does  the  output  conform 
with  what  was  required?  Verification  is  performed  at  each  phase  and  between 
each  phase  of  the  development  lifecycle.  It  determines  that  each  phase  and 
subphase  product  is  correct,  ccxnplete,  and  consistent  with  itself  and  with  its 
predecessor  product.  Testing,  either  automated  or  manual,  examines  program 
behavior    by    executing    the   program  on  sample  data  sets,  e.g.    passing  data. 
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autcraatically  or  manually,  through  a  design  to  determine  the  correctness  of 
the  program  is  testing  the  design. 

V,V&T  is  commonly  used  as  a  single  expression;  this  is  not  to  infer  that  the 
methodology  is  an  all  or  nothing  process.  Project  constraints,  suda  as 
criticality,  error  tolerance  or  budget,  should  determine  hew  much  validation, 
verification,  or  testing  is  applied  to  that  project.  This  docunent  uses  the 
general  term,  V,V&T,  referring  to  the  total  methodology;  it  is  understood 
that  the  reader  will  extract  those  portions  of  V,V&T  which  are  feasible  and 
applicable  to  the  specific  situation.  V,V&T  may  be  performed  by  an 
independent  V,V&T  group  (IV&V),  by  the  person(s)  producing  the  product  or  a 
combination  of  both.  Again,  the  decision  as  to  who  performs  the  V,V&T  is 
project  dependent.  The  objective  of  V,V&T  is  to  ensure  the  correctness, 
completeness,  and  consistency  of  the  final  product. 

Software  V,V&T  focuses  on  the  prevention  and  the  detection  of  errors,  i.e. 
deviation  from  intent.  This  is  accomplished  throu^  the  use  of  both  manual 
and  automated  techniques.  Errors  include  deficiencies  such  as  unsatisfied 
requirements,  or,  the  converse,  the  inclusion  of  extraneous  functions.  An 
error  may  be  in  the  coding  of  the  software,  a  specification  of  the  software, 
(e.g.,  a  design  specification)  or  the  docunentation  (e.g.,  a  user's  manual). 
The  error  might  be  related  to  the  functional  correctness  or  another  property, 
such  as  performance,  or  a  more  subjective  attribute  such  as  product  form. 

To  describe  the  role  of  V,V&T  in  error  detection  and  prevention,  three 
categories  of  V,V&T  activities  are  described.  These  are  illustrated  in  Figure 
2.1-1. 

CHECK  INTEGRITY  CHECK  EVOLUTION 


SPECIFICATION 

REFINEMENT  ^ 

SPECIFICATION 

ELABORATION  ^ 

CHECK  APPROPRIATENESS/SUFFICIENCY 


Figure  2.1-1    Three  Categories  of  V,V&T  Activities 


The  objective  of  integrity  checking,  the  first  category  of  V,V&T  activities, 
is  to  verify  the  integrity  of  the  products  at  each  phase  of  development.  Each 
product  is  analyzed  for  internal  consistency  and  completeness.  For  example,  a 
requirements  specification  can  be  analyzed  to  detect  inconsistent  or 
contradictory  requirements  such  as  the  specification  of  an  output  report  that 
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requires  data  which  is  unavailable. 

The  objective  of  evolution  checking,  the  second  category,  is  to  ensure  the 
completeness  and  consistency  between  levels  of  specification,  where  the  second 
is  a  refinement  or  elaboration  of  the  first. 

The  objective  of  appropriateness^sufficiency  checking,  the  third  category,  is 
to  compare  this  evolving  solution  against  the  problem  as  currently  understood 
to  ensure  that  it  is  a  necessary  and  sufficient  solution. 

2.2  A  PROBLEM  SOLVING  MODEL  FOR  SOFTWARE  DEVELOPMENT 


DEVELOPMENT  PHASE 

Requirements  Definition  and  Analysis  Subf*iase 

0  Development  of  the  Project  V,V&T  Plan 
0  Generating  Requirements  Based  Test  Cases 
o  Review  and  Analysis  of  the  Requirements 

Preliminary  Design  Subphase 

o  V ,V&T  Planning 

o  Generating  Design  Based  Test  Scenarios 
0  Review  and  Analyze  the  Preliminary  Design 

Detailed  Design  Subp*iase 

0  Generation  of  Design  Based  Functional  Test  Data 
0  Review  and  Analysis  of  the  Detailed  Specification 

Progranming  and  Testing  Sub^iase 

o  Ccanplete  the  Test  Case  Specification 

o  Review,  Analysis,  and  Testing  of  the  Program 

Installation  Subp*iase 

o  System  Acceptance 

OPERATION  AND  MAINTENANCE  PHASE 

o  Software  Evaluation 

o  Software  Modification  Evaluation 

o  Regression  Testing 


Figure  2.2-1       Stanmary  of  V,V&T  Activities 
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The  following  discussion  describes  each  phase/ sub phase  by  presenting  the 
nature  and  purpose  of  each,  the  products  produced,  and  the  V ,V&T  activities 
which  incrementally  build  confidence  in  the  software  product  as  it  evolves. 
Figure  2,2-1  simraarizes  the  V,V&T  activities. 

2.2.1  INITIATION  PHASE 


Description:  The  initiation  phase  begins  with  the  recognition  of  a  problem 
and  concludes  with  a  decision  of  whether  or  not  to  implement  a  software 
solution.  The  recognition  of  a  problem  may  be  sudden  or  gradual;  the  problem 
may  be  generally  recognized  or  only  perceived  by  a  small  group.  The  main 
activity  of  the  initiation  phase  is  a  joint  exploration  of  the  problem  by  the 
group  having  the  problem  and  the  group  responsible  for  solving  it.  The 
decision  to  pursue  a  solution  must  be  based  upon  a  clear  understanding  of  the 
problem,  a  preliminary  investigation  of  alternative  solutions,  and  a 
comparison  of  the  expected  benefits  versus  the  cost  (design,  construction,  and 
operation)  of  the  solution. 

Products:  The  primary  result  is  a  decision  about  the  continuation  of  the 
project.  To  support  this  decision,  there  are  often  three  products.  These 
are: 

0  The  project  request  or  proposal:    defines  the  problem  to  be  solved  and  the 
scope  and  objectives  of  the  proposed  solution. 

o  The  feasibility  study:    states  the  assunptions  being  made,  defines 
alternative  solutions  and  assesses  technical  and  operational  feasibility. 

o  The  cost/benefit  analysis:    estimates  and  compares  the  costs  (construction 
and  operation)  and  the  expected  benefits  (both  quantifiable  and  intangible) . 

V,V&T  Activities:  The  three  products  are  reviOf^ed  by  the  problem  posers,  the 
problem  solvers,  and  the  decision  makers  (e.g.,  higher  level  management  in 
control  of  required  resources).  The  following  questions  are  generally 
answered  in  this  review. 

o  Has  the  scope,  cost,  impact,  and  urgency  of  the  problem  been  adequately 
defined? 

o  Has  a  technically  feasible  and  operationally  viable  solution  been 
identified? 

0  Have  alternative  solutions  been  identified? 
o  Have  alternatives  been  adequately  studied? 

o  Have  the  costs  of  the  solutions  been  analyzed  and  compared  against  the 
expected  benefits?    What  are  the  probabilities  of  achieving  these  benefits? 
What  are  the  risks  involved? 
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2.2.2  DEVELOPMENT  PHASE 

As  an  example,  this  phase  is  divided  into  five  subi*iases  for  the  purpose  of 
illustration.    These  are  each  separately  described. 


2.2.2.1  REQUIREMENTS  DEFINITION  AND  ANALYSIS  SUBPHASE 

Description:  The  goal  of  the  requirements  subfAiase  is  to  put  the  problem  into 
a  rigorous  form  upon  which  a  solution  can  be  based  i.e. ,  a  stat«nent  of  the 
requirements  which  a  software  solution  must  satisfy.  Requirements 
identification  is  iterative,  involving  the  problem  posers  and  the  problem 
solvers.  Requirements  may  be  modified  in  later  subp^iases  as  a  better 
understanding  of  the  problem  is  gained.  These  modifications  are  docunented, 
creating  a  traceable  record  of  the  progress  and  evolution  of  the  final 
product.  Also  during  this  subf^ase,  two  planning  activities  are  performed. 
First,  project  plans,  budgets,  and  schedules  for  the  develoment  phase  and  each 
subphase  are  developed.  In  addition,  the  V,V&T  goals  are  identified  and  a 
plan  for  achieving  these  goals  is  developed. 

Products:  This  subf^ase  results  in  the  preparation  of  four  products.  The 
first  three  are  completed,  while  the  fourth  is  finished  during  subsequent 
phases. 

o  The  software  requiranents  docunent:  Specifies  what  the  systen  must  do. 
This  includes  the  requisite  information  flows  and  processing  functions. 
Acceptance  criteria  for  deciding  that  the  requirements  are  satisfied 
as  specified  is  an  important  part  of  this  product, 

o  The  project  plan:    Specifies  a  strategy  for  managing  the  software 
development.    It  defines  the  goals  and  activities  for  all  phases  and 
subphases.    It  includes:    resource  estimates  over  time  and  intermediate 
milestones  including  management  and  technical  reviews.    It  defines 
methods  for  design,  docimentation,  problem  reporting,  and  change  control. 
It  also  specifies  supporting  techniques  and  tools, 

o  The  Validation,  Verification,  and  Testing  Plan:    Specifies  goals  of  the 
V,V&T  activities  including  software  testing.    It  is  the  design  of  a 
project  specific  V,V&T  process  and  identifies  techniques  and  tools  to 
assist  in  achieving  the  goals.  It  specifies  plans  (schedules,  budgets, 
responsibilities,  etc.)  for  performing  theV,V&T  activities. 

o  The  Software  Test  Case  Specification:    Describes  the  scenarios  and  cases 
for  testing  each  requirement.    The  acceptance  criteria  are  used  to  develop 
test  cases.    Expected  results  for  each  test  case  are  included. 

V,V&T  Activities:    There  are  three  basic  thrusts  to  these  activities. 

These  are: 

o  Development  of  the  Project  V,V&T  Plan:    Goals  for  V,V&T  activities  are 
determined;  a  V,V&T  process  is  designed;  techniques  and  tools  are 
chosen;  schedules  and  budgets  are  established. 
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o  Generating  Requirements  Based  Test  Cases:  These  form  a  basic  set  of  test 
cases.  Th^  help  clarify  and  determine  the  measurability  of  the  software 
requirements  and  form  a  basis  for  acceptance  testing. 

o  Review  and  Analysis  of  the  Requirements:    The  goal  is  to  ensure  that  the 
requirements  identified  will  result  in  a  feasible  and  a  usable  solution  to 
the  entire  problem.    They  are  reviewed  for  clarity,  completeness, 
correctness,  consistency,  testability,  and  traceability  to  the  problem 
statement. 


2.2.2.2  PRELIMINARY  DESIGN  SUBPHASE 

Description:  During  preliminary  design,  the  problem  solvers  (the  software 
developers)  assisted  by  the  problem  posers  (the  custoner/user)  formulate  and 
analyze  alternative  solutions.  This  may  reveal  flaws  in  the  requirements  and 
result  in  its  modification.  This  iteration  continues  until  all  issues  have 
been  resolved. 

This  subphase  results  in  a  high  level  specification  of  the  solution.  The 
solution  is  conceptual  in  nature,  defining  information  aggregates,  information 
flows,  and  logical  processing  steps.  It  will  describe  all  the  major 
interfaces,  and  their  inputs  and  outputs.  Implementation  details,  e.g., 
actual  programs  and  physical  data  structures  are  generally  not  addressed. 

Project  plans  (schedules,  budgets,  deliverables,  etc.)  are  reviewed  and 
revised  as  appropriate  for  the  scope  and  complexity  of  the  solution 
formulated. 

Products:  There  is  one  new  product  of  this  sublease  -  the  preliminary  design 
specification.  In  addition,  each  of  the  four  products  of  the  requirements 
subphase  may  undergo  revision  or  be  supplemented  with  neti  data. 

o  The  Preliminary  Design  Specification:  Docunents  the  high  level  solution 
developed  during  this  phase.    This  may  be  packaged  in  two  separate 
docunents,  i.e. ,  a  systoi/ subsystem  specification  and  a  data/database 
specification. 

o  A  Revised  Requirements  Specification:    Design  activities  may  reveal 
inconsistent,  infeasible,  or  ambiguous  requirements  and  result  in  the 
revision  of  their  specification. 

o  An  Updated  Project  Plan:    Upon  the  ccmpletion  of  the  preliminary  design, 
the  scope  and  complexity  of  the  solution  are  well  understood.    As  a  result, 
the  project  plan  (schedules,  budgets,  deliverables,  etc.)  is  made  more 
accurate  and  realistic. 

o  An  Updated  V,V&T  Plan:    This  plan  may  warrant  revision  based  upon  new  or 
revised  requirements. 

o  Software  Test  Case  Specification:    Additional  test  scenarios  and  test  cases 
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are  developed  to  exercise  and  test  aspects  of  the  design. 

V,V&T  Activities:    There  are  three  areas  of  V,V&T  activity: 

o  V,V&T  Planning:    The  V,V&T  plan  is  reviewed  and  revised  as  deemed 
necessary. 

o  Generating  Design  Based  Test  Scenarios:    Complementing  and  expanding 
the  requirements  based  test  data  generated  focusing  on  the  logical 
functions  performed. 

o  Review  and  Analyze  the  Preliminary  Design:    To  assure  internal 

consistency,  completeness,  correctness  and  clarity;  to  verify  that  the 
design  is  linked  to  and,  when  implemented,  will  satisfy  the  requirements. 

2.2.2.3  DETAILED  DESIGN  SUBPHASE 

Description:  The  purpose  of  this  subphase  is  to  refine,  resolve  deficiencies, 
define  additional  details,  and  package  the  logical  solution  created  in  the 
previous  subphase.  Implementation  details  are  addressed.  Ambiguities  are 
removed  from  the  design  specification.  The  detailed  design  specification 
describes  the  physical  solution  (algorithms  and  data  structures)  which  is  an 
elaboration  of  the  logical  solution  specified  in  the  preliminary  design.  The 
result  is  a  solution  specification  that  can  be  implemented  in  code  with  little 
or  no  need  for  additional  analysis. 

The  detailed  design  team  may  discover  processing  operations  that  are 
iirpractical  or  impossible  to  implement,  necessitating  modification  of  the 
preliminary  design  and  possibly  the  requirements  as  well. 

The  user  and  participants  in  the  review  are  consulted  in  making  major  design 
decisions. 

Products:  There  are  two  new  products  of  this  subphase  and  additional 
information  added  to  an  existing  one: 

o  The  Detailed  Design  Specification:    A  fully  detailed  description  of  the 
software  (algorithms  and  data)  to  be  coded  in  the  following  subphase. 

o  Software  Test  Case  Specification:    This  is  now  a  substantially  complete 
description  of  test  data  and  the  expected  results. 

o  Problem  Reports:  Formal  statements  of  observed  problems.  This  may 
necessitate  going  back  to  a  previous  subphase. 

V,V&T  Activities:    The  two  V,V&T  activities  of  this  phase  are: 

o  Gemration  of  Design  Based  Functional  Test  Data:    Formulated  test  data 
based  on  the  physical  structure  of  the  syst«n. 

o  Review  and  Analysis  of  the  Detailed  Specification:    To  assure  internal 
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consistency,  completeness,  correctness,  and  clarity;  to  verify  that  the 
detailed  design  is  linked  to  and  is  a  correct  refinement  of  the  preliminary 
design;  to  validate  that  the  design  when  implemented  will  satisfy  the 
requiranents. 


2.2.2.4  PROGRAMMING  AND  TESTING  SUBPHASE 

Description:  This  subphase  results  in  a  program  which  is  ready  for 
installation.  Programming  is  the  process  of  implementing  the  detailed  design 
specification  into  code.  Only  minor,  if  any,  design  issues  are  resolved 
during  this  subphase. 

Ccxnpleted  code  undergoes  testing  as  described  in  the  V  ,V&T  plan.  Generally, 
three  types  of  testing  are  performed:  unit,  integration,  and  systan.  While 
unit  testing  is  done  by  the  programmer,  the  person(s)  responsible  for 
integration  and  system  testing  is  project  specific. 

Unit  testing  checks  for  typographical,  syntax,  and  logic  errors.  Each  of  the 
modules  of  code  are  checked  individually  by  the  programmers  who  wrote  th«n  to 
ensure  that  each  correctly  implements  its  design  and  satisfies  the  specified 
requirements. 

Integration  testing  focuses  on  checking  the  intermodule  communication  links, 
and  testing  aggregate  functions  formed  by  groups  of  modules. 

System  testing  examines  the  operation  of  the  syst«n  as  an  entity.  This  type 
of  testing  ensures  that  the  software  requirements  have  been  satisfied  both 
singly  and  in  ccwibination. 

The  final  activity  of  this  subphase  is  planning  the  installation  of  the 
software. 

Products:  There  are  six  new  outputs  produced  during  this  subpiiase  and  one 
other  which  is  canpleted. 

o  Software  Test  Case  Specification:    Final  revisions  and  additions  to  the 
test  data  are  made. 

o  Program  Code:    Fully  docunented  and  tested  code,  which  is  ready  for 
installation. 

o  Test  Results  and  Test  Evaluation  Reports:    The  documentation  of  the 
comparison  of  actual  and  expected  results. 

o  User  Docunentation:    Manuals  describing  the  input  and  report  formats, 
user  canmands,  error  messages,  and  instructions  for  operation  by  the 
user. 

o  Maintenance  Manual:  Docunentation  to  maintain  the  system. 

o  Installation  Plan:    Specifies  the  approach  to  and  details  of  the 
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installation  of  the  software, 

o  Problem  Reports:  Formal  statenents  of  observed  problems.  This  may 
necessitate  going  back  to  a  previous  sub^iase. 

V,V&T  Activities:    The  V,V&T  activity  of  this  phase  focuses  upon  the  program 

produced, 

o  Complete  the  Test  Case  Specification:    Final  additions  and  modifications 
necessary  due  to  design  changes  made  during  coding, 

o  Review,  analysis  and  testing  of  the  program:    Includes  checking  for 
adherence  to  coding  standards,  manual/ autcxnated  analysis  of  the  program, 
and  the  execution  of  the  program  on  test  data  ensuring  that  it  meets 
the  acceptance  criteria. 

2,2.2.5  INSTALLATION  SUBPHASE 

Description:  The  result  of  this  subphase  is  a  system  incorporating  the 
developed  programs,  other  software  components,  the  hardware,  and  production 
data.  The  activities  of  the  subphase  are  guided  by  the  installation  plan 
developed.  The  first  task  is  to  integrate  the  system  components.  Integration 
consists  of  installing  hardware,  installing  the  programCs)  onto  the  computer, 
refonnatting/cr eating  the  data  base(s),  and  verifying  that  all  canponents  have 
been  included.  Modification  to  program  code  may  be  necessary  to  obtain 
compatibility  between  hardware  and  software,  or  between  different  software 
modules. 

The  next  task  is  to  test  the  system.  The  test  data  fran  earlier  subphases  is 
enhanced  and  used  here.  The  result  is  a  systen  qualified  and  accepted  for 
production  use. 

The  ttiird  task  is  the  start  of  syst«n  operation.  The  strategies  for  this 
include  ininediate  cutover,  phased  cutovers,  or  parallel  operation.  This  task 
also  includes  operator  and  user  training. 

Products:  The  new  product  of  this  subphase  is  the  Installation  Report,  but 
previously  completed  products  may  be  updated  to  incorporate  findings  of  the 
installation  activity. 

o  Installation  Report:    Describes  the  results  of  the  installation  activities, 
including  data  conversion,  installation  testing/results,  and  software  system 
problems  and  modifications  necessary. 

V,V&T  Activities:  The  primary  V,V&T  activity  centers  on  acceptance  of  the 
syston  by  the  custcmer.  This  could  be  a  simple  statanent  signed  by  a  customer 
representative.    This  marks  the  end  of  the  development  phase. 
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2.2.3  OPERATION  AND  MAINTENANCE  PHASE 

Description:  The  final  phase  involves  the  actual  use  of  the  software  and 
monitoring  its  operation  to  ensure  that  it  succeeds  in  solving  the  user's 
problem. 

Most  often,  sane  need  for  modifying  the  software  arises  during  this  phase. 
The  maintenance  process  involves  determining  the  reason  for  each  modification. 
IVie  cause  could  be  an  error  made  in  the  original  development,  the  recognition 
of  a  new  requirement,  or  the  desire  for  a  design  modification  to  improve 
performance,  usability,  etc.  Once  the  cause  is  determined,  the  software  (code 
and  documentation)  is  'redeveloped*  from  that  point.  For  example,  the 
redevelopment  due  to  a  change  in  requironents  will  result  in  modifications  to 
the  requironents  specification,  the  design,  the  code,  and  user  and  operation 
manuals. 

Problon  reporting,  change  requests,  and  other  change  control  mechanisms  are 
used  to  facilitate  the  systematic  correction  and  evolution  of  the  software. 
In  addition,  performance  measurement  and  evaluation  activities  are  performed 
to  ensure  that  the  syst^  continues  to  meet  its  requirements  in  the  context  of 
a  changing  system  environment. 

Products:  To  track  and  manage  the  evolution  of  the  software  in  this  post 
development  phase,  several  new  outputs  are  produced: 

o  Problem  reports:  Formal  statements  of  observed  problems.  The  analyses 
of  these  may  result  in  software  change  requests. 

o  Change  requests:    Requests  for  specific  modification  to  the  software. 
These  could  be  generated  due  to  an  error  (i.e.,  problem  report)  or  a 
modification  of  the  requirements  or  design. 

o  Revision  to  initial  development  products:    As  a  result  of  change  requests 
any  one  or  all  of  the  products  of  the  initiation  and  development  phases  may 
require  revision. 

V,V&T  Activities:    The  V,V&T  activities  of  this  phase  can  be  separated  into 
two  categories:    the  monitoring  and  evaluation  of  the  syston,  and  the  problem 
correction  activities.    The  first  is  an  on-going  activity.    The  second  is 
driven  by  the  problem  reports  and  change  requests, 

o  Software  evaluation:    Activities  aimed  at  assessing  the  operation  of  the 
software  and  assuring  continued  satisfaction  of  user  requironents, 

o  Software  modification  evaluation:    As  needs  for  change  are  discovered  and 
requests  for  modifications  are  made,  the  requested  modifications  are 
evaluated  in  the  same  manner  as  the  original  software  is  evaluated. 
After  modifications  are  made,  the  changes  are  revia^ed  and 
tested  to  ensure  that  the  expected  changes  in  the  software  is  achieved. 

0  Regression  testing:  Rerunning  test  cases  which  a  program  has  previously 
executed  correctly  in  order  to  detect  errors  created  during  software 


correction/modification  activities. 
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CHAPTER  THREE 

A  FRAMEWORK  FOR  INTEGRATED  VALIDATION,  VERIFICATION,  AND  TESTING 

To  do  effective  V,V&T,  the  development  team  needs  to  select  a  well-matched, 
compatible  set  of  techniques  and  tools.  The  selection  is  based  on  the  V,V&T 
goals  of  the  project.  These  goals  take  into  account  the  characteristics  of 
problem  and  the  constraints  on  the  solution.  From  the  user's  perspective,  the 
techniques  and  tools  must  ensure  functional,  efficient,  reliable  and  lew 
maintenance  software.  The  developer  is  concerned  with  these  issues  as  well  as 
the  integrity  and  compostion  of  the  software.  This  blend  of  concerns  is  the 
guiding  force  in  selecting  V,V&T  techniques  and  tools.  This  chapter  presents 
information  which  assists  in  the  selection  of  a  complementary  set  of 
techniques  and  tools  and  their  application  throu^out  developnent. 
Specifically,  it  will  discuss  the  following: 

Types  of  analyses  that  are  performed  -  description  of  static,  dynamic,  and 
formal  analyses, 

An  integrated  approach  to  performing  V ,V&T  -  combining  the  three  types  of 
analysis  in  a  complementary  fashion  to  achieve  a  V,V&T  technology,  and 

Application  of  the  VyV&T  approach  -  the  use  of  the  three  types  of  analysis  in 
applying  V,V&T  to  requirements,  design,  and  code. 

3.1  STATIC  ANALYSIS 

The  static  analysis  detects  errors  throu^  examination  of  the  product,  Figure 
3.1-1.  Some  examples  of  the  errors  detected  are:  language  syntax  errors, 
misspellings,  incorrect  punctuation,  improper  sequencing  of  statonents,  and 
missing  specification  elements.  Static  analysis  techniques  may  be  manually  or 
automatically  applied,  however,  automated  techniques  require  a  machine 
readable  product. 


PRODUCT/SPECIFICATION 
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FORM  AND 
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REPORTS 
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-► 

DIAGNOSTICS 

Figure  3.1-1    General  Form  of  Static  Analysis 

Manual  static  analysis  techniques  may  be  applied  to  all  development  products, 
e.g.,  requirements  statement,  program  code,  or  a  users  manual.  In  general, 
they  are  straightforward  and,  when  applied  with  discipline,  are  effective  in 
preventing  and  detecting  errors. 
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The  application  of  certain  manual  techniques,  such  as  desk  checking, 
inspection,  and  walkthrougjis,  provide  certain  advantages  over  the  use  of 
specialized  automated  techniques.  One  advantage  is  that  different 
perspectives  can  be  addressed  simultaneously.  A  product  may  be  examined  for 
high  level  as  well  as  detailed  properties.  Another  advantage  is  that  manual 
analysis  provides  an  opportunity  for  the  analyst  to  apply  various  heuristics, 
i.e.  aids  to  discover  errors,  and  subjective  judgements.  A  general  weakness 
of  the  manual  techniques  is  that  correct  usage  often  involves  tedious  and 
repetitious  activities.  As  the  size  of  the  application  increases,  the 
tendency  is  to  compromise  on  the  thorough  application  of  the  technique  which 
results  in  an  increasing  chance  of  error. 

Automated  static  analysis  tools  most  often  operate  on  program  source  code  but 
can  operate  on  requirements  and  design.  Two  kinds  of  static  analyzers  can  be 
identified.  The  first  gathers  and  reports  information  about  a  program.  These 
kinds  of  analyzers  generally  do  not  search  for  any  particular  type  of  error  in 
a  program.    A  symbol  cross  reference  generator  is  an  example  of  this  type. 

The  second  kind  of  analyzer  detects  specific  classes  of  errors  or  anomalies  in 
a  progran.  Examples  of  this  type  include:  (1)  parsers  which  determine  the 
adherence  of  a  program  to  the  language  syntax;  they  may  include  additional 
local  progranming  conventions/standards  such  as  program  length;  (2) 
techniques  for  analyzing  the  consistency  of  actual  and  formal  parameter 
interfaces  (see  Figure  3.1-2);  (3)  techniques  for  comparing  all  variable 
references  with  their  declarations  to  check  for  consistency;  and  (4) 
techniques  for  analyzing  a  program  for  erroneous  sequences  of  events  or 
operations,  for  example,  attonpting  to  read  from  a  file  before  it  is  opened. 

A  much  more  detailed  description  of  many  specific  techniques  and  tools  can  be 
found  in  the  Technique  and  Tool  Reference  Guide[POWE]. 


Procedure 
Mismatch 


Figure  3.1-2  Module  Interface  Consistency  Checks 
3.2  DYNAMIC  ANALYSIS 


Dynamic  analysis  is  the  process  of  detecting  errors  through  the  stucfy  of  the 
response    of   a    program   to   a  set  of  input  data.    It  is  usually  accomplished 
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through  an  autcmated  simulation  or  the  execution  of  a  program,  but  may  be 
manually  performed,  e.g.,  a  walkthrough.  Dynamic  analysis  (see  Figure  3.2-1 ) . 
is  the  processes  of: 

o  preparing  for  test  execution, 

o  test  execution,  and 

o  analysis  of  test  results. 

Preparing  for  test  execution  includes  test  data  preparation  and  formulation  of 
expected  results. 


SPECIFICATION  OF  FUNCTTONAL  INTENT 
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Figure  3.2-1    General  Form  of  Dynamic  Analysis 


Test  data  preparation  involves  the  formulation  of  test  scenarios,  test  cases 
and  the  data  which  are  to  be  entered  into  the  program.  Test  scenarios  and 
test  cases  are  chosen  as  the  result  of  analyzing  the  requirements  and  design 
specifications  and  the  code  itself.  The  test  data  should  demonstrate  and 
exercise  externally  visible  functions,  program  structures,  data  structures, 
and  internal  functions  including  impossible  and  improbable  test  cases.  Each 
test  case  includes  a  set  of  input  data  and  the  expected  results.  The  expected 
results  may  be  expressed  in  terms  of  final  values  and  as  statonents 
(assertions),  embedded  in  the  software,  about  intermediate  states  of  program 
execution. 

The  development  of  assertions  takes  place  during  the  design  and  programming 
subphases  of  the  lifecycle.  In  general,  assertions  are  stat«nents  that 
specify  the  intent  of  a  program's  behavioral  properties  and  constraints. 
Assertions  about  inputs,  outputs,  and  intermediate  steps  of  each  function  may 
be  generated.  A  special  notation  (an  assertion  language)  often  is  used  to 
specify  the  assertions.  Assertions  are  developed  and  inserted  into  the  actual 
design  specification  and  program  code,  usually  as  specially  formatted 
comments. 

Currently,  test  preparation  is  accomplished  primarily  throu^  manual  methods. 
These  include  definition  of  specification  based  functional  tests  and 
cause-effect  graphing,  a  test  case  design  methodology.  Specification  based 
functional  testing  is  a  method  of  developing  test  scenarios,  test  data  and 
expected  results  throu^  the  examination  of  the  program  specifications  (in 
particular,    the  requirements,  design  and  program  code).    Test  scenarios  based 
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upon  the  requirements  have  the  objective  of  demonstrating  that  the  functions, 
performance  and  interface  requirements  and  solution  constraints  are  satisfied. 
Test  cases  are  determined  from  the  design  to  test  functions,  structures, 
algorithms  and  other  elements  of  the  design.  Test  data  are  determined  from 
the  program  to  exercise  computational  structures  implemented  in  the  program 
code. 

Cause-effect  grappling  [MYER]  is  a  technique  to  develop  test  cases  based  upon 
inputs  and  input  conditions.  For  each  case,  the  expected  outputs  are 
identified.  This  technique  operates  on  the  requirements  and  design 
specifications. 

Test  execution  involves  running  a  program  with  the  prepared  test  cases  and 
then  collecting  the  results.  Testing  may  be  planned  and  performed  in  a 
top-down  or  botton-up  fashion  or  a  combination  of  the  two,  Top>-down  testing 
is  performed  in  parallel  with  top-down  construction  in  that  a  module  is 
developed  and  tested  while  sutmodules  are  left  incomplete  as  stubs  or  dunmy 
routines,  Bottcra-up  testing  consists  of  testing  pieces  of  code,  individual 
modules,  and  small  collections  of  modules  in  that  order,  before  they  are 
integrated  into  the  total  program.  Bottom-up  testing  may  require  the 
development  of  test  driver  routines. 

Test  execution  may  be  performed  manually.  Design  and  code  walkthrougjris  (i,e, , 
a  manual  simulation)  provide  a  straightforward  dynamic  analysis  method  used 
prior  to  execution  and  during  debugging. 

Instrumentation,  the  insertion  of  code  into  a  program  to  measure  program 
characteristics,  may  be  required  to  assist  in  testing.  It  may  be  done  to 
capture  intermediate  values  of  a  computation,  measure  execution  frequencies 
during  testing,  or  to  detect  assertion  violations.  Instrumentation  may  be 
done  manually,  or  automatically  as  with  test  coverage  analyzers  and  dynamic/ 
assertion  processors. 

Four  types  of  tools  are  commonly  used  to  assist  in  test  execution.    They  are: 

o  Test  coverage  analyzers, 

o  Execution  monitors, 

o  Dynamic  assertion  processors,  and 

o  Performance  monitors. 

Test  coverage  analyzers  capture  and  report  execution  details  (e.g,,  statanent 
execution  counts).  Scxne  test  tools  (often  built  into  a  compiler  or  runtime 
system)  monitor  execution  and  check  for  the  adherence  to  certain  language 
semantics  (e.g.,  array  subscripts  must  remain  within  declared  bounds,  divisors 
must  be  non-zero).  E^namic  assertion  processors  monitor  program  execution  and 
report  violations  of  the  assertions  supplied  by  the  analyst.  Performance 
monitoring  tools  report  execution  data  describing  execution  frequency  and 
timing  information. 

The  process  of  analyzing  the  outputs  of  testing  involves  canparing  the  actual 
to  expected  results.  This  analysis  requires  a  specification  of  the  expected 
results  for  each  case.    Comparison  of    actual    and    expected    results  may  be 
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performed  manually,  or  if  the  data  are  machine  readable,  an  autanated 
comparator  may  be  used.  The  detection  of  assertion  violations  is  normally 
accomplished  through  analyzing  the  assertion  results  generated  by  the 
instrumented  program. 

Dynamic  analysis  also  includes  determining  the  thoroughness  of  the  testing. 
Commonly  used  metrics  are  the  nunbers  (or  percentages)  of  executable 
statements,  branches,  and  paths  actually  exercised  by  the  tests.  Each 
succeeding  metric  subsunes  the  others.  Since  the  nunber  of  paths  grows 
exponentially  with  the  nunber  of  decision  points,  even  small  programs  can  have 
many  thousands  of  paths.  For  this  reason,  attempting  to  ensure  that  a  high 
percentage  of  paths,  exclusive  of  loop  iterations,  is  exercised  can  be  very 
costly.  The  statanent  and  branch  coverage  metrics,  althou^  less  conclusive, 
are  most  frequently  used  because  they  are  relatively  easy  and  inexpensive  to 
implement. 

3.3  FORMAL  ANALYSIS 

Formal  analysis  involves  the  use  of  rigorous  mathonatical  techniques  to 
analyze  the  algorithms  of  a  solution.  Algorithms  are  analyzed  for  nunerical 
properties  (e.g.  accuracy,  stability,  convergence),  efficiency,  and 
correctness.  At  present,  formal  analysis  is  primarily  a  manual  activity  with 
limited  autanated  assistance. 

For  more  detailed  information  about  techniques  and  tools  for  measurement  of 
testing  in  the  analysis  of  the  numerical  properties  of  algorithms,  two  types 
of  analysis  can  be  identified.  The  first  type  makes  a  conjecture  about  an 
algorithm's  nunerical  properties  and  then  proves  a  theorem  to  establish  the 
conjecture.  In  the  second  type,  an  automated  tool  is  used  to  analyze  the 
nunerical  stability  of  a  sequence  of  conputations. 

Complexity  analysis  first  makes  conjectures  about  the  nunber  of  operations 
and/or  the  amount  of  space  used  on  a  problem  of  arbitrary  but  fixed  size.  The 
next  step  is  to  construct  arguments  which  support  the  conjectures.  There  is 
considerable  discussion  as  to  how  to  measure  or  express  software  complexity. 
This  technique  has  not  reached  maturity. 

In  another  type  of  formal  analysis,  two  basic  approaches  are  used.  One 
approach  is  to  prove  correctness  of  a  whole  program.  The  second  approach  is 
to  prove  correctness  of  particular  program  properties. 

The  basic  strategy  to  both  approaches  is  to  construct  a  set  of  reasoning  that 
shows    that  a  solution  specification  satisfies  its  requiranent(s) .  Typically, 
this  is  done  by  comparing    the   inferred    transformations   to    the  functional 
transformations  dictated  by  the  specifications  of  intent  (see  Figure  3.3-1). 

To  do  the  comparison,  symbolic  execution  using  selected  control  paths  through 
the  algorithmic  specification  is  employed.  For  each  path  the  values  of  each 
data  object  encountered  are  computed.  Each  value  is  not  computed  as  a  nunber 
but  rather  as  a  function  or  formula.  Inputs  are  not  assigned  specific  values 
but  rather  are  treated  as  unquantified  variables.  As  a  result,  at  the  end  of 
tracing    down    the    selected   path,  the  functional  relation^ips  of  outputs  to 
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inputs  are  determined.  These  functions  or  formulas  are  then  compared  to 
specifications  of  functional  intent  to  see  if  the  solution  specification  is 
correct  as  a  value  transformer. 
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Figure  3.3-1    General  Form  of  a  Formal  Functional  Analysis 

The  formal  analysis  of  a  specific  path  can  ensure  that  the  functions  computed 
in  traversing  the  path  are  correct,  and  thus  that  any  input  processed  by 
traversal  of  that  path  will  necessarily  be  transformed  into  correct  output 
values. 

Formal  analysis  has  two  major  limitations:  strict  dependence  on  the  validity 
of  the  assimptions  on  which  the  analysis  is  based,  and  the  complexity  of  the 
pathwise  analysis  of  even  anall  programs.  Formal  analysis  has  not  yet  reached 
its  full  potential  as  a  V,V&T  technique  for  several  reasons.  For  one,  program 
specifications  are  rarely  written  with  sufficient  precision  to  permit  a 
rigorous  canparison  of  intent  with  the  implemented  program.  Also, 
accomplishing  the  analysis  manually  is  extremely  tedious  and  difficult  (thus 
prone  to  error)  and  very  few  automated  tools  are  currently  available  to 
facilitate  effective  application  of  formal  techniques, 

3.4  AN  INTEGRATED  APPROACH  TO  V,V&T 

Each  of  the  three  types  of  analysis  —  static,  dynamic,  and  formal  —  provides 
the  V,V&T  analyst  with  different  types  of  specific  information  about  the 
solution  being  examined.  Static  analysis  focuses  on  the  form  and  structure  of 
the  solution,  but  not  the  functional  or  canputational  aspects.  Dynamic 
analysis  addresses  the  functional,  structural,  and  computational  aspects.  It 
is  used  to  detect  errors  relating  to  these,  but  in  practice  is  not  used  to 
dononstrate  the  absence  of  errors  and  is  limited  in  demonstrating  correctness. 
Formal  analysis  can  provide  a  strong  statement  regarding  certain  properties  of 
a  solution  including  correctness,  but  is  limited  by  the  difficulty  of 
application  and  lack  of  autanated  support. 

These  three  types  of  analysis  can  be  integrated  to  not  only  achieve  the 
benefits  of  each,  but  to  be  complementary  and  provide  a  powerful  conposite 
V,V&T  technology. 

This  section  briefly  describes  a  strategy  to  achieve  this  integration  (see 
Figure  3.4-1).  The  following  three  sections  of  this  chapter  describe  hew  this 
strategy  can  be  applied  to  software  requirements  and  design  specifications, 
and  to  the  code  itself. 
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Figure  3.4-1    General  V,V&T  Integration  Strategy 

The  integration  strategy  is  simple.  First,  static  analysis  techniques  and 
tools  are  applied  to  analyze  the  form  of  the  specification.  These  techniques 
and  tools  are  straight  forward  to  apply,  applicable  to  all  levels  of 
specification,  and  identify  flaws  preventing  the  application  of  dynamic  and 
formal  techniques. 

The  second  step  is  the  application  of  dynamic  analysis  techniques  and  tools. 
These  focus  on  the  functional  meaning  of  the  solution  and  detect  errors  in 
their  specification.  These  may  be  manually  applied  to  the  requirements  and 
design  specifications,  with  the  code  undergoing  dynamic  testing.  This 
process,  when  applied  with  discipline,  is  effective,  comprehensive  and  within 
the  resojrce  constraints  of  nearly  all  projects. 

If  additional  assurances  are  required,  the  third  step  is  the  application  of 
formal  analysis  techniques.  These  techniques  can  give  strong  assurances  that 
the  progran  design  and  code  are  fully  traceable  to  the  requirements  and  that 
they  are  a  necessary  and  sufficient  solution  to  the  stated  problem. 

3.5  REQUIREMENTS  V,V&T 

Figure  3.5-1  illustrates  the  integrated  V,V&T  approach  for  requirements 
specification.  It  involves  the  application  of  static  analysis  techniques  and 
tools  to  inspect  the  form  and  structure  of  the  requirements  and  dynamic 
analysis  techniques  and  tools  to  examine  their  functional  aspects.  For 
requironents  specification,  the  definition  of  dynamic  analysis  is  extended  to 
include  all  actions  which  examine  behavioral  and  performance  aspects  of 
requiranents.  Static  and  dynamic  analysis  techniques  and  tools,  whether 
strictly  manual,  automated,  or  a  combination  of  both  are  generally  applicable 
to  the  four  types  of  requirements: 

o  functional  processing  requirements. 
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o  performance  requirements, 
o  interface  requirements,  and 
o  solution  (design)  constraints. 

Static  techniques  and  tools  are  most  useful  for  analysis  of  product  form  and 
interface  requirements  and  solution  constraints,  while  dynamic  analysis  (such 
as  simulation)  is  most  useful  for  performance  requirements. 
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Figure  3.5-1  Integrated  Approach  to  Requiranents  V,V&T 

Static  analysis  normally  focuses  on  checking  adherence  to  specification 
conventions,  completeness  and  language  syntax.  Dynamic  analysis  at  this  level 
normally  focuses  upon  information  flows  and  functional  interrelationships. 
Manual  methods  such  as  inspections,  peer  reviews,  and  walkthrou^s  are 
effective  in  accanplishing  both  types  of  analyses  if  rigorously  performed.  An 
example  of  using  a  manual  method  and  dynamic  analysis  is  to  include  input  data 
as  part  of  a  walkthrou^. 

The  application  of  both  types  of  analysis  depends  upon  the  method  used  to 
specify  the  requirements.  If  the  constructs  of  the  specification  schane  are 
clearly  defined  and  capable  of  being  represented  in  a  computer  processable 
form,  then  automated  tools  to  aid  in  performing  both  the  static  and  dynamic 
analysis  may  be  used.  Several  such  specification  methods  with  supporting 
tools  are  in  existence,  but  not  widely  used  or  available. 

The  requirements  specification  methods  commonly  in  use  today  support  the 
representation  of  information  aggregates  and  functional  capabilities  in  a 
hierarchical  form.  This  allows  for  two  types  of  checks  of  the  specification 
for  internal  consistency.  The  first  examines  the  decompositions  performed  in 
creating  the  functional  and  information  hierarchies.  Each  level  of 
decanposition  is  checked  to  ensure  that  it  is  consistent  with  the  previous. 
The  second  type  of  analysis  examines  the  consistency  between  the  functional 
and  information  views.  Information  aggregate  specifications  usually  include 
indications  of  the  functions  which  generate  and  utilize  them.  Conversely, 
function  specifications  usually  include  indications  of  the  information 
aggregates  which  they  generate  and  utilize.    These  two  types  of  specifications 
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are  readily  checked  for  consistency. 

The  dynamic  analysis  of  a  specification  often  involves  the  evaluation  of  the 
functional  aspects  of  the  solution  with  respect  to  the  intent  as  docunented  in 
previous  specifications.  This  is  not  possible  because  the  requirements 
statement  is  the  first  formal  specification.  The  only  previous  docunent  to 
which  it  might  be  compared  is  the  project  request  from  the  initiation  phase. 
Normally,  this  is  an  insufficient  basis  for  requirements  analysis. 

Consequently,  the  dynamic  analysis  of  requirements  is  acccmplished  throu^  the 
examination  of  expected  functional  behavior  to  determine  if  it  will  solve  the 
problem.  Exercising  information  transformations  throu^  plausible  sequences 
of  the  required  functions  will  provide  insight  to  the  expected  behavior.  This 
is  usually  guided  by  scenarios  which  describe  expected  use,  including  input 
values  and  expected  results.  (These  scenarios  form  the  basis  for  the  software 
test  data  and  are  refined  and  complemented  in  the  design  and  coding 
subphases.)  This  type  of  analysis  aids  in  the  recognition  of  errors, 
oversights,  and  contradictions  in  the  requirements. 

3.6  DESIGN  V,V&T 

Figure  3.6-1  illustrates  the  integration  strategy  as  applied  to  a  design 
specification.  Similar  to  the  requirements  analysis,  static  analysis 
techniques  and  tools  are  applied  to  analyze  the  form  and  structure  of  the 
design  specification(s) .  Dynamic  analysis  techniques  and  tools  are  used  to 
examine  the  functional  aspects  of  the  design.  In  addition,  formal  analysis 
may  be  applied  to  rigorous  design  specifications  to  gain  additional  assurances 
of  correctness  relating  to  certain  functional  properties. 
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Figure  3.6-1    Integrated  Approach  to  Design  V,V&T 

The  basis  for  design  V,V&T  is  the  design  docunentation.  The  mechanisms  or 
design  representation  schemes  used  to  specify  the  design  determine  the 
specific  analysis  techniques  which  can  be  employed.  The  degree  of  formality 
of  the  schemes  used  determines  the  need  for  and  ability  to  perform  static 
analysis.  The  content  of  the  information  captured  by  the  scheme  determines 
the    dynamic    and   formal    analysis    techniques  which  may  be  applied.    If  the 
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schanes  can  be  translated  into  a  computer  processable  form,  then  automated 
techniques  may  be  used. 

Syntactic  and  s«nantic  errors  can  be  detected  by  the  static  analysis  of  a 
design  specification.  The  syntactic  errors  are  errors  in  form  rather  than 
content.  These  are  not  the  primary  emphasis  of  static  analysis.  By  assuring 
the  correct  form  and  structure  of  the  design  specification,  the  way  is  cleared 
for  more  in  depth  analysis  of  the  semantic  content  of  a  specification. 

The  semantic  errors  which  can  be  detected  in  a  design  involve  information  or 
data  decomposition,  functional  decomposition,  control  flew,  and  data  flow. 
Selected  examples  of  these  are  discussed  belcw. 

Design  specification  schemes  generally  provide  mechanisms  for  specifying 
algorithms  and  their  inputs  and  outputs  in  terms  of  modules.  Various 
inconsistencies  in  specifying  the  flow  of  data  objects  throu^  the  modules  are 
possible.  For  example,  a  module  may  need  a  particular  data  object  which  no 
other  module  creates.  Conversely,  a  module  may  create  a  data  object  which  is 
not  input  to  any  module.  Static  analysis  can  be  applied  to  detect  these  types 
of  data  flow  errors. 

Certain  errors  made  during  the  composition  of  a  design  can  also  be  detected. 
Design  specifications  are  usually  created  by  iteratively  supplying  detail. 
Thus,  most  schemes  facilitate  the  hierarchical  expression  of  a  design.  Data 
aggregates  and  functional  modules  may  be  specified  in  terms  of  their  gross 
overall  characteristics  and  then  specified  in  more  detail.  A  hierarchical 
specification  structure  is  regarded  as  an  excellent  vehicle  for  expressing  and 
understanding  a  design.  It  does,  however,  leave  open  the  possibility  of 
inconsistencies  between  levels  of  detail.  For  example,  the  inputs  and  outputs 
specified  for  a  high  level  module  must  be  equivalent  to  the  cimulative  inputs 
and  outputs  of  the  sutraodules.  Any  inconsistencies  indicate  an  error  in  the 
evolving  solution.  Static  analysis  can  determine  the  presence  or  absence  of 
such  errors. 

Dynamic  analysis  of  a  design  is  generally  accomplished  by  some  form  of  design 
simulation.  This  may  be  a  manual  walkthrou^  or  simulation  using  a  model  of 
the  design.  A  design  walkthrou^  is  similar  to  the  analysis  performed  on  the 
requirements  except  at  a  greater  level  of  detail.  It  is  guided  by  usage 
scenarios  which  are  refined  and  expanded  from  those  used  in  the  requirements 
analysis.  At  the  design  stage,  test  cases  are  developed  and  used  during  the 
walkthrough  to  exercise  all  functions.  During  detailed  design,  test  cases  are 
developed  and  used  to  examine  the  software  structure  as  well  as  its  functions. 
(These  test  scenarios  and  test  cases  are  used  during  the  programming  subphase 
for  testing  the  code.)  Manual  walkthrou^s,  when  rigorously  performed  and 
guided  by  documented  test  scenarios,  are  a  cost  effective  technique  for 
analyzing  a  software  design. 

For  larger  software  designs  and  highly  critical  systems  or  components,  an 
automated  simulation  may  be  appropriate.  This  requires  the  construction  and 
execution  of  a  solution  model  with  the  test  scenarios.  The  model  is  validated 
as  a  faithful  representation  of  the  solution.  The  cost  of  simulation 
generally  increases  with  the  complexity  of  the  model  and  the  degree   of  model 
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fidelity.    Thus,  mcxlel  simulation  is  only  used  when  it  can  be  cost  justified. 

Fonnal  analysis  techniques  may  be  manually  applied  to  a  design  specification 
This  involves  tracing  paths  throu^  the  design  specification  and  formulating  a 
composite  function  for  each.  This  procedure  is  more  feasible  at  higher  levels 
of  a  hierarchical  design  specification  because  less  detail  is  present, 
resulting  in  algorithm  paths  being  relatively  short  and  few  in  nunber.  Thus, 
the  evolved  functions  remain  concise  and  manageable. 

The  purpose  for  deriving  these  ccmposite  functions  for  a  given  level  of  design 
is  computed  and  compared  to  the  functions  of  the  previous  level.  This  process 
assures  that  the  design  is  continuing  to  specify  the  same  functional  solution 
as  it  is  hierarchically  elaborated. 

The  formal  analysis  of  a  design  specification  can  be  improved  by  using 
autcraated  symbolic  execution  tools.  Such  tools  can  be  expensive  to  create  and 
operate.  In  return,  however,  they  offer  greater  speed  and  capacity  for 
manipulating  detailed  specifications.  Thus,  the  functional  effects  of  all 
levels  of  a  design  specification  can  be  determined. 

Unfortunately,  such  soi*iisticated  tools  are  rarely  applied  in  contemporary 
practice  principally  due  to  the  novelty  of  such  tools,  the  infrequent  use  of 
rigorous  design  formalisms,  and  expense.  It  is  likely,  as  experience  with 
these  tools  is  gained  and  as  better  design  representation  schemes  emerge, 
their  use  will  increase. 

3.7  CODE  V,V&T 

The  third  and  last  application  of  the  general  V,V&T  integration  strategy  is  to 
program  code.  This  is  portrayed  in  Figure  3.7-1 .  The  V ,V&T  procedure  for 
code  includes  botti  static  and  dynamic  analysis.  It  may  also  enccmpass  formal 
analysis. 

Because  code  is  written  to  be  canpiled  and  executed,  it  is  necessarily  written 
in  a  axnputer  language  having  defined  syntax  and  s«nantics.  Syntax  rules  are 
generally  enforced  througji  the  use  of  a  compiler.  Unfortunately,  most 
compilers  do  not  carry  out  checking  for  many  important  semantic  errors,  e.g. 
array  range  subscript  checking,  type  analysis  or  routine  interface  analysis. 

Static  analysis  techniques  and  tools  are  used  to  ensure  the  proper  form  of 
programming  products  such  as  code  and  docunentation.  This  can  be  accomplished 
by  checking  adherence  to  coding  and  docunentation  conventions,  interface  and 
type  checking,  etc.  The  checking  can  be  done  by  manual  techniques  such  as 
inspections  and  automated  tools  such  as  a  code  auditor. 

Next,  dynamic  analysis  techniques  are  employed  to  stu(fy  the  functional  and 
canputational  correctness  of  the  code.  Initially,  manual  techniques  such  as 
walkthroughs  can  be  used  as  an  effective  forerunner  to  testing.  These 
techniques  can  focus  on  modules  to  complement  the  role  of  testing. 
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Figure  3.7-1    Integrated  Approach  to  Code  V,V&T 

Testing  is  accomplished  by  running  the  code  on  the  test  data  sets  which  were 
developed  during  the  requirements  and  design  subp*iases.  As  emphasized 
earlier,  the  correctness  of  the  test  executions  is  determined  more 
definitively  when  the  expected  results  are  specified.  Testing  for  adherence 
to  assertions  is  also  highly  advisable.  The  assertions,  which  are  products  of 
the  design  activity,  provide  additional  information  regarding  expected 
behavior  of  the  software. 

In  cases  where  software  is  being  developed  in  an  envirorment  other  than  the 
production  envirorment,  testing  is  more  problematical.  Here  the  production 
envirorment  can  be  simulated  or  taken  into  account  informally.  In  ary  case, 
the  validity  of  the  test  results  depends  upon  the  fidelity  of  the  simulation 
or  informal  judgements.  If  there  is  a  significant  difference  in  the  two 
envirorments,  there  will  be  an  eventual  need  for  some  additional  testing  in 
the  actual  production  envirorment.  The  balance  between  simulation  testing  and 
actual  production  envirorment  testing  must  be  determined  for  each  individual 
project,  based  partially  upon  the  availability  and  expensiveness  of  the 
production  environment. 

If  assurances  of  correctness  over  and  above  those  provided  by  dynamic  analysis 
are  required,  then  formal  analysis  should  follow  testing.  For  most  projects, 
this  simply  takes  the  form  of  inspections  to  see  that  the  various  algorithms 
dictated  by  the  design  have  been  correctly  implemented.  Coming  after  a 
battery  of  successful  tests,  this  activity  needs  to  focus  upon  algorithms 
which  are  deemed  crucial  and  yet  inadequately  tested  (perhaps  due  to  high  cost 
of  exercising  them).  Of  course,  seme  applications  may  be  particularly 
critical  in  nature  and  demanding  of  very  great  assurance  of  correctness. 
Symbolic  evaluation  and  other  formal  analysis  methods  can  be  effective  in 
achieving  such  levels  of  confidence,  but  the  cost  is  high,  generally  entailing 
development  of  special  purpose  evaluation  tools. 
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3.8  SUMMARY 

It  is  reasonable  to  conclude  that  for  any  software  project,  the  integration 
strategy  for  V,V&T  activities  will  residt  in  a  better  return  on  investment  of 
costly  resources  and  increased  confidence  in  the  desired  qualities  of  the 
final  product.  Although  the  strategy  is  sound  and  is  strongly  endorsed,  it 
does  not  address  the  problem  of  how  to  select  and  configure  specific 
techniques  and  tools  for  a  specific  project.  The  problems  in  doing  this 
appear  to  be  of  two  major  types: 

o  Projects  vary  in  many  respects  and  this  variation  strongly  affects  the 
proper  selection  of  techniques  and  tools,  and 

o  There  are  a  multiplicity  of  techniques  and  tools  but  there  is  little 
guidance  in  their  selection  to  meet  specific  project  needs. 

The  first  problem  is  addressed  in  Chapter  4  of  this  docunent.  The  second 
problem  is  addressed  in  part  by  the  report  "Features  of  Software  Development 
Tools"  [HOUGa]  and  the  "NBS  Software  Tools  Database"  [HOUGbj. 

In  time,  a  reasonable  complement  of  proven  V ,V&T  support  techniques  and  tools 
will  evolve.  Then,  with  a  judicious  selection  and  systanatic,  integrated 
application  during  all  lifecycle  phases,  it  will  be  possible  to  effect 
significant  reductions  in  the  cost  of  software  production  as  well  as 
inproveraents  in  the  quality  of  developed  systems. 
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CHAPTER  FOUR 
V,V&T  PLANNING 

An  important  part  of  problem  solving  is  planning.  This  is  certainly  true  in 
software  development.  Within  this  context,  V,V&T  planning  is  an  integral  part 
of  the  overall  project  development  planning.  It  is  started  during  the 
initiation  of  a  project  along  with  all  other  planning. 

Ihe  objective  is  to  establish  a  V,V&T  plan  to  suit  the  needs  of  the  project. 
Software  development  and  maintenance  projects  vary  with  respect  to  many 
factors  such  as  size,  criticality,  canplexity,  etc.  These  factors  must  be 
taken  into  account  so  that  the  plan  is  both  feasible  and  effective. 

This  chapter  describes  the  contents  of  a  V,V&T  plan  and  presents  a  process  for 
developing  one.  Section  4.1  explains  how  V,V&T  planning  is  part  of  the 
overall  planning  process.  It  briefly  describes  the  outcome  of  a  V,V&T  plan 
and  introduces  a  four  step  process  to  assist  in  developing  part  of  the  plan. 
The  remaining  sections  present  each  of  the  four  steps. 

Step  I  -  Identify  V^V&T  Goals  -  The  result  is  a  set  of  specific  measurable 
goals  of  the  validation,  verification,  and  testing  activities  of  the  project. 

Step  II  -  Determine  Influences  on  V  ^V&T  Activities  -  The  result  is  a  set  of 
factors  to  be  taken  into  account  in  the  planning  of  the  V,V&T  activity. 

Step  III  -  Select  V,V&T  Techniques  and  Tools  -  The  outcome  is  a  list  of  V,V&T 
practices,  techniques  and  tools  to  satisfy  the  identified  goals  within  the 
constraints  of  the  envirorment. 

Step  IV  -  Develop  a  Detailed  V,V&T  Plan  -  The  result  is  a  phase-by-phase  V,V&T 
plan  specifying  V,V&T  practices,  as  well  as  specific  tasks,  schedules,  and 
budgets. 

4.1  V,V&T  PLANNING  IN  THE  TOTAL  PROJECT  CONTEXT 

Project  planning  is  one  of  the  major  functions  of  project  management.  V,V&T 
planning  is  one  part  of  project  planning.  Figure  4,1-1  illustrates  this 
relationship  and  examples  of  planning  products. 

The  objective  of  planning  is  to  docunent  a  plan  of  action.  It  may  include: 
objectives,  approach,  schedule,  and  allocated  resources.  A  plan  is  used  to 
initiate  and  control  a  project. 

Figure  4.1-2  shows  the  evolution  of  a  project  plan.  A  project  is  formed  to 
solve  a  problem.  This  first  step  in  project  formation  is  to  define  its 
mission  or  its  objectives.  Next,  an  approach  is  formulated.  This  is  refined 
with  additional  details  about  the  approach  and  docunented  in  the  project  plan. 


Page  29 


V,V4T  PLAN  EXAMPLES: 


S/W  DEVELOPMENT  PLAN 
QUALITY  ASSURANCE  PLAN 
CHANGE  CONTROL  PLAN 
INSTALLATION  PLAN 
TRAINING  PLAN 
DOCUMENTATION  PLAN 

Figure  Software  Project  Planning 


Problem  ►  Mission  ►Approach  ►Plan 


Figure  4,1-2     Evolution  of  Project  Plan 

This  chapter  presents  a  V,V&T  planning  process  which  follows  a  similar  set  of 
steps.  The  interactions  between  the  general  and  theV,V&T  planning  activities 
are  important.  The  general  planning  activity  will  drive  the  V,V&T  planning 
activity,  which  in  turn  will  provide  input  and  feedback  to  the  first.  Some  of 
the  important  interactions  are: 

o  V,V&T  goals  are  established  which  complement  the  general  project  approach, 
o  V,V&T  techniques  and  tools  will  assist  in  achieving  the  V,V&T  goals  only 

if  they  are  integrated  with  the  project  approach, 
o  The  details  of  the  V,V&T  plan,  e.g.,  time  and  resource  requirements,  must  be 

factored  into  the  overall  schedule  and  budgets. 

Planning  activities,  including  the  preparation  of  the  V,V&T  plan,  may  begin  in 
the  initiation  phase  but  are  completed  early  in  the  development  phase  (Figure 
4.1-3).  Plans  specifying  tasks,  budgets,  schedules,  etc.  for  the 
requirements  and  preliminary  design  subpiiases  are  completed  early  in  the 
subphase.  Plans  specifying  information  for  the  subsequent  subphases  are  also 
prepared  to  the  level  of  detail  possible.  These,  though,  are  likely  to  be 
revised  as  the  technical  scope,    size    and   complexity    of    the    solution  are 
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defined  in  tJie  preliminary  design  activity.    Special  attention  must  be  paid  to 
activities  that  require  long  lead  time,  or  need  to  begin  early  in  the  project, 
e.g.,  tool  acquisition  or  development,  personnel  hiring  or  training. 


Initiation 


Development 


Initial 
Plans 


Revised  Plans 
(as  necessary) 


^  ^ 


Figure  4.1-3       Pr^aration  of  Plans 


The  result  of  the  V ,V&T  planning  activity  is  a  docunented  V ,V&T  plan.  The 
ccxnplexity  of  the  plan  and  the  effort  required  to  prepare  it  will  depend  upon 
the  size  and  nature  of  the  project.  More  effort  and  a  greater  level  of  detail 
may  be  required  as  the  size,  complexity,  and  critical  nature  of  the  project 
increases. 


A  V,V&T  plan  specifies  goals  of  the  project's  V,V&T  activities  relating  to  the 
products  of  each  phase.  For  these  products,  the  activities  and  supporting 
techniques  and  tools  are  then  described  which  make  up  the  approach  to  achieve 
the  goals.  An  abbreviated  outline  of  a  V,V&T  plan  is  presented  in  figure 
4.1-=4.    A  detailed  outline  is  presented  in  Section  4.5. 

The  remainder  of  this  chapter  describes  the  four  step  approach  to  preparing  a 
V,V&T  plan.  Each  step  and  substep  are  presented  in  a  standard  format, 
including  the  following  items  as  appropriate: 

o  General  Description 
o  Outputs 
0  Inputs 

o  Interrelation^ips  with  Other  Steps 

0  Roles  and  Responsibilities 

o  Method  and  Supporting  Techniques 

o  Worksheet 

o  Example 

o  Conments 

The  worksheets  are  suggested  as  docunentation  aids.  Examples  are  presented 
using  these  worksheets. 

For  anall  projects  it  may  be  sufficient  to  abbreviate  the  process,  following 
it  as  necessary  to  achieve  the  ultimate  objective,  a  V,V&T  plan.  For  larger 
projects,  the  process  provides  a  systematic  method  for  developing  a 
comprehensive  V,V&T  plan.  In  either  case,  the  outcome  should  suit  the  goals 
of  the  project. 


I.  BACKGROUND  AND  INTRODUCTION 

II.  V,V&T  GOALS 

1,  Summary  of  V,V&T  Goals  and  Measurement  Criteria 

2.  References  and  Related  Docunents 

III.  PHASE  BY  PHASE  PLANS 

For  each  phase,  information  including: 

1 .  V,V&T  Goals  for  Each  Product 

2.  V,V&T  Activities 

3.  Techniques  and  Tools 

4.  Assumptions  and  Other  Information 

5.  Roles  and  Responsibilities 

6.  Schedules 

7.  Budgets 

8.  Personnel 


Figure  4.1-M 
ABBREVIATED  OUTLINE  OF  A  PROJECT  V,V&T  PLAN 
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4.2  STEP  I:    IDENTIFYING  V,V&T  GOALS 

General  Description:  V,V&T  goals  are  formulated  fron  software  requirements, 
and,  secondarily,  software  product  standards.  V,V&T  goals  address  the 
functional,  structural,  and  computational  correctness  of  the  software,  the 
correctness  of  the  form,  its  performance,  and  other  attributes  such  as 
reliability  and  portability.  The  achievonent  of  these  goals  danonstrates  that 
the  software  has  the  desired  attributes.  This  is  straightforward  in  seme 
instances,  e.g.,  demonstrating  the  correct  operation  of  a  tape  sort/merge 
capability,  while  in  others  it  may  be  a  matter  of  degree,  e.g.,  clarity  and 
understandability  of  docunentation.  The  term  goal  describes  a  specific, 
measurable  outcome.  A  complete  definition  of  a  goal  includes  the  statement  of 
the  measurement  criteria  to  determine  that  the  goal  has  been  reached. 

Goals  are  established  to  define  tangible  results  of  the  activity  (e.g.,  a 
report)  or  a  culminating  event  (e.g.,  a  review).  Measurement  criteria  specify 
hew  to  determine  that  the  result  is  satisfactory. 

Goals  relating  to  products  (intermediate  and  final)  are  established  which 
pertain  to  either  their  form  or  the  content.  The  goal  will  precisely  describe 
the  desired  product  attributes.  This  distinction  will  be  useful  in  selecting 
techniques  and  tools. 

Goals  relating  to  an  activity  or  product  should  be  examined  as  a  set,  and 
categorized  as  high,  medium,  or  lew  in  importance.  This  information  will  be 
used  in  St^  III  if  trade-offs  (because  of  time,  budget,  or  resource 
constraints)  are  necessary.  This  examination  also  acts  as  a  re-evaluation 
point,  and  may  result  in  the  revision,  elimination,  or  combination  of  goals. 
The  worksheet  examples  for  steps  I  and  III  are  separate.  To  eliminate 
duplication,  they  can  be  combined  into  one  worksheet. 

Outputs  of  Step  I: 

o  A  set  (one  or  more)  of  goals,  measurement  criteria,  and  importance 
of  each  V,V&T  goal 

Inputs  to  Step  I: 

o  Project  Proposal 

o  Statement  of  Problen/Needs 

o  Statonent  of  Software  Requirements 

o  Applicable  Product  Standards 

Interrelationships  with  Other  Steps:  The  list  of  goals  frera  this  step  will  be 
used  in  Step  III.    This  step  can  be  done  parallel  with  Step  II. 

Roles  and  Responsibilities: 

o  (J ser/Cu sterner  -  to  identify  the  goals  and  measurement  criteria  to  ensure 
that  the  product  is  acceptable 

o  Development  Staff  and  Management  -  to  ensure  that  V,V&T  goals  and 
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measurement  criteria  are  both  measurable  and  feasible  and  reflect 
existing  standards  and  conventions  concerning  product  quality 

Methods  and  Supporting  Techniques: 

o  Standards,  guidelines  and  conventions  that  contribute  to  product  quality 

o  Requirement  analysis  and  specification  aids  that  support  the 
identification  and  documentation  of  the  software  requirements 

o  Checklists  for  identifying  V,V&T  goals  based  upon  the 
software  requirements 

0  Inspection,  walkthrou^,  review  procedures  that  support  the  analysis 
and  evaluation  of  the  V ,V&T  goals 

V,ViT  GOALS  WORKSHEET 

PROJECT  :^  

General  Product; 
Activity/ 

Product  Goal  Measurement  Criteria  Importance 


Product:  All  critical  modules 

Software  will  undergo 

thorough  testing 


Test  data  will  be  generated 
to: 

1)  Demonstrate  the  adherence 
to  functional  specification, 
including  nominal  values  and 
extremes  of  the  domain  and  range. 

2)  Exercise  all  module  branches, 

3)  Traverse  all  recovery  paths, 
i.e.,  an  execution  path  where 

an  error  (e.g.,  Illegal  input  data) 
is  discovered  and  for  which  the 
system  must  continue  to  operate. 
(These  paths  must  be  identified 
In  the  program  internal  and 
external  documentation). 


high 


Figure  4.2-1    Goals  and  Measurement  Criteria 


Example:  Figure  U.2-1  identifies  the  goal,  measurement  criteria,  and 
importance  of  testing  critical  modules. 

Comments:  A  worksheet  which  assists  in  specifying  goals  and  selecting  the 
associated  techniques  and  tools  are  described  in  the  following  sections.  It 
can  be  used  as  an  aid  in  docunenting  the  results  of  this  step. 

o  V,V&T  Goals  Worksheet  -  used  in  Step  I  to  define  a  specific  set  of  goals 
and  the  measurement  criteria 
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o  V,V&T  Technique  and  Tool  Selection  Worksheet  -  used  in  Step  3  to  select 
the  techniques  and  tools  to  attain  each  goal 
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4.3  STEP  II:    DETERMINE  FACTORS  INFLUENCING  V,V&T  ACTIVITIES 

General  Description:  This  step  involves  collecting  a  mixture  of  both 
objective  and  subjective  information  (e.g.,  the  attitudes  of  the  technical 
staff  concerning  V,V&T).  This  information  will,  in  seme  cases,  lead  to  clear 
cut  decisions  regarding  the  implementation  of  a  given  technique  or  tool.  In 
other  instances,  it  may  only  indicate  potential  areas  of  difficulty  or  supply 
information  to  support  a  decision  intuitively. 

Four  general  areas  in  a  suggested  order  of  investigation  are  presented  belcw. 

Software  V,V&T  and  development  technology 
Project  groups  and  roles 
Project  constraints 
Available  computing  resources 

These  areas  of  investigation  are  further  described  belcw. 

Outputs  of  Step  II: 

o  An  assessment  of  available  software  engineering  and  V,V&T  technology 
0  The  capabilities  and  expected  involvement  of  various  groups 
o  The  budget  and  schedule  constraints  upon  the  V,V&T  activities 
o  An  inventory  of  computing  resources. 

Inputs  to  Step  II: 

o  Software  engineering  standards,  guidelines,  procedures,  development  and 

maintenance  metiiodologies,  etc. 
o  Information  regarding  technical  personnel 
o  Information  regarding  project  budget  and  sdiedule 
o  Information  regarding  computing  resources 

Interrelationships  with  Other  Steps:  This  step  can  be  performed  in  parallel 
with  Step  I  -  The  Identification  of  V,V&T  Goals.  It  provides  inputs  into  Step 
III  and  Step  IV. 

Roles  and  Responsibilities:  This  activity  is  primarily  the  responsibility  of 
the  development  organization. 

Methods  and  Supporting  Techniques:  The  following  is  a  checklist  of  the  four 
areas  of  investigation  mentioned  above. 

Software  V,V&T  and  Development  Technology 

o  Software  V,V&T  (see  techniques  and  tools  descriptions  in  [POWE]) 

-  techniques 

-  tools 

-  standards  and  guidelines 

-  conventions  and  procedures 

o  Software  development 
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-  methodologies 

-  specification  techniques  and  design  representation  schemes 

-  techniques  and  tools 

-  docunentation  standards  and  guidelines 

-  technical  assistance 

-  training 

Project  Groups  and  Roles  (The  following  groups  and  their  expected  roles 
should  be  balanced  against  their  knowledge,  experience,  attitudes,  and 
expectations.) 

o  Customer 
o  User 

o  Technical  staff 

o  Management  (e.g.,  project,  or  organizational,  and  support,  e.g., 
legal) 

o  Independent  groups  (e.g.,  quality  assurance,  independent  V ,V&T) 

Project  Constraints 

0  Schedule  of  major  project  results 
0  Budget  information  for: 

-  analysis  and  review  of  each  product 

-  acquisition  and/ or  development  of  support  capabilities  and  tools 

-  technical  assistance  and/or  training  in  the  use  of  selected 
techniques  and  tools 

-  use  of  independent  testing,  V,V&T  and  quality  assurance  teams 

Available  Computing  Resources 

0  Available  machines 

o  Access  methods  and  devices 

0  Suppcrt  utilities 

0  Technical  support 
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4.4  STEP  III:    SELECT  V,V&T  TECHNIQUES  AND  TOOLS 

General  Description:    This  step  can  be  divided  into  two  activities: 

o    Establish  a  candidate  list  of  techniques  and  tools  to  be  used  to  reach 
each  goal,  and 

o    Select  and  evaluate  the  techniques  and  tools  from  the  candidate  list 
to  be  used. 

One  input  to  the  first  activity  is  the  list  of  V,V&T  goals.  Goals  should  have 
been  established  for:  a)  activities  by  phase/ subphase,  and  b)  the  form  and 
content  of  each  product.  The  other  input  to  this  activity  is  information 
about  each  candidate  technique  or  tool. 

Candidate  techniques  and  tools  may  be  chosen  from  three  sets:  a)  those 
commonly  used  on  software  development  projects  (information  on  these  was 
collected  during  Step  II);  b)  those  used  elsewhere  which  may  be  acquired; 
and  c)  those  which  can  be  developed  by  the  project  staff. 

For  each  candidate  technique  or  tool,  sane  basic  information  is  needed.  This 
includes  the  availability  of  docunentation,  training,  consulting  support,  and 
tool  effectiveness.  Also  required  are  cost,  time  required  for  acquisition, 
and  need  for  training  project  personnel. 

Once  the  list  of  candidate  techniques  and  tools  has  been  created  for  a  given 
goal,  the  selection  activity  begins.  The  feasibility  of  implementing  each 
technique  or  tool  is  studied.  The  implementation  requirements  (e.g., 
personnel  resources  and  level  of  expertise)  are  compared  against  the 
constraints  identified  in  Step  II  (e.g.,  personnel  experience).  This 
identifies  inappropriate  candidates.  The  techniques  and  tools  to  be  used  are 
chosen  from  the  remaining. 

Ihe  assumptions  made  in  the  selection  process  need  to  be  documented  for  use  in 
Step    IV.    For  example,  assunptions  regarding  the  acquisition,  implementation, 
application,  and  required  training  should  be  recorded. 

Outputs  of  Step  III: 

o  A  list  of  V,V&T  techniques  and  tools 
o  Assumptions  made  in  selection  process 

Inputs  to  Step  III: 

o  V  ,V&T  goals 

0  Information  on  candidate  tools  and  techniques 

o  Influences  to  be  considered  during  selection  (from  Step  II) 

Interrelationships  with  Other  Steps:  The  list  of  V,V&T  goals  may  be  revised 
because  one  or  more  goals  are  determined  to  be  infeasible. 
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Roles  and  Responsibilities:  The  development  staff  have  primary  responsibility 
for  this  step.  It  should  be  performed  by  senior  staff  familiar  with  both  the 
management  and  the  technical  aspects  of  the  project.  Generally,  a 
representative  of  the  groups  who  will  use  the  techniques  and  tools  should  be 
involved  in  the  selection  process. 

Methods  and  Supporting  Techniques:  The  "Validation,  Verification,  and  Testing 
Technique  and  Tool  Reference  Guide"  [POWE]  describes  individual  techniques  and 
tools. 

Worksheet  and  Example:  Figure  4.4-1  shews  the  V,V&T  Technique  and  Tool 
selection  worksheet.  Figure  4.4-2  shows  a  completed  worksheet  for  a  goal  from 
the  previous  example. 


1 


V.ViT  TECHNIQUE  AND  TOOL  SELECTION  WORKSHEET 


PROJECT: 


Goals 


Measurement 
Criteria 


Candidate  Techniques 
&  Tools 


Rationale  for 
Choice/Elimination 


Final  Choice 
&  Comments 


The  goals  and  criteria 
established  during 
Step  I. 


Techniques  or  tools  which  Each  technique  or  tool 

may  potentially  be  used  will  be  analyzed  and 

in  reaching  each  goal  are  either  chosen  or 

listed  in  this  column  rejected.  The  decision 


Is  documented  here. 


The  final  cholceCs) 
will  be  listed  here. 
Also  any  Information 
pertaining  to  the 
use  or  application 
should  be  stated  here 
or  reference  to  supple- 
mentary material 
given.    It  is  Important 
to  note  here  any  special 
information  that 
should  be  taken  into 
account  in  the  detailed 
planning  (Step  IV) 
and/or  implementation 
of  the  technique  or 
tool. 


Figure  4.4-1    V,V&T  Technique  and  Tool  Selection  Worksheet 


Page  39 


V.VtT  TECHNIOUE  t  TOOL  SELECTION  VWJRKSIIEET 
PROJECT t  


Goals 

For  critical 
f jnctions I 
All  critical 
fflrxJulcs  will 
undergo 
thorough 
testing 


Measurement 
Criteria 

The  functions  of 
these  modules 
raust  be  tested 
for  extremal 
valuen  ot  the 
domain  and 
range,  as  well 
as  on  nominal 
test  datas 
1)  All  branches 
will  be  tested, 
including  all 
' recovery ' 
paths.  I.e. 
an  execution  path 
where  an  error 
(e.g.,  Illegal 
input  data)  Is 
discovered  and 
tot  which  the 
syst»m  must 
continue  to 
opf'tat'!  (these 
paths  must  be 
identified  in 
the  program 
Internal  and 
external 
documentation) . 


Candidate  Techniques 
t  Tools 

-Data  Flow  Analyser 


-Assertion  Generation 
and  Assertion 
Processing 


-Specification  Based 
Functional 
Testing 


-Test  Coverage 
Analyzer 


Rationale  for 
Choice/Elimination 

Can  be  used  to  check  out 
programs  from  the  data 
flow  perspective. 
Availability  is  uncertain. 

Assertion  generation  will 
aid  in  specification 
documentation  and  test 
generation.  Assertion 
checking  via  an  autimatic 
processor  will  aid  in 
testing  and  test  coverage 
measurement . 

Availability  is  uncertain. 

This  technique  can  be 
used  in  the  requirements 
and  design  phases  to 
incrementally  build  test 
sets. 

This  tool  can  summariae 
test  coverage  to  enable 
an  accurate  assessment 
of  the  thoroughness  of 
the  testing. 


Final  Choice 
t  Comments 

Invest Iqate 
availability 


Investigate 
availability 


Used  in  conjunction 
with  test  set  docu- 
mentation standards 
and  test  libraries  to 
store  data  and  results. 

Will  be  usf?d. 
Modifications  to 
existing  tool  will 
will  be  necessary. 


Figure  4.4-2  Completed  V,V&T  Technique  and  Tool  Selection  Worksheet 
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4.5  STEP  IV:    DEVELOP  A  DETAILED  V,V&T  PLAN 

General  Description:  The  preceding  three  steps  provide  a  definition  and  means 
of  attaining  V,V&T  goals.  The  detailed  V,V&T  plan  can  now  be  formulated  by 
defining  each  activity  in  terms  of  specific  tasks  and  outccmes.  The  roles  and 
responsibilities  of  the  groups  involved,  (e.g.,  customer,  user,  technical, 
management)  are  defined;  sign-off  procedures  at  milestones  are  established. 
Underlying  assunptions  are  stated,  and  interrelation^ips  with  other 
activities  (preceding,  parallel  with,  and  succeeding)  are  defined.  Budget 
allocations  and  schedules  are  established.  Plans  also  include  training  and 
tool  acquisition  or  development  activities. 

Outputs  of  Step  IV: 

o  A  Detailed  V,V&T  Plan 

Inputs  to  Step  IV: 

o  List  of  V,V&T  techniques  and  tools 

o  Project  Schedule,  budget,  and  personnel  information 

o  V  ,V&T  goals 

o  Technique  and  tool  selection  assunptions 

Interrelationships  with  Other  Steps:  The  plan  is  built  on  the  outputs  of 
prior  steps. 

Roles  and  Responsibilities:  This  involves  both  management  and  technical 
staff.  Management  approval  is  required  for  schedules  and  budgets. 
Oust  oners/ users  approve  their  role. 

Methods  and  Supporting  Techniques:  Presented  below  is  an  annotated  outline  of 
a  general  V,V&T  plan.    This  can  be  used  as  a  checklist. 

An  Outline  of  a  Project  V,V&T  Plan 

I.  Background  and  Introduction 

The  purpose  of  this  section  is  to  establish  the  context  for  the 
docunent.  It  should  be  brief  and  introductory  in  nature.  It 
should  focus  on  those  aspects  of  the  problem  and/or  solution  which 
are  the  sources  of  the  goals  for  V,V&T.  Where  additional 
information  is  necessary,  it  can  be  included  or  referenced. 

A.  Statement  of  Problem 

B.  Proposed  Solution 

C.  Project  Summary 

D.  References/Related  Docunents 

II.  V,V&T  Goals  and  Measurement  Criteria 
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This  section  presents  the  results  of  Step  I.  It  presents  the 
project  V,V&T  goals,  measurement  criteria,  and  importance.  The 
exact  format  and  content  of  this  section  may  vary.  If  the 
worksheets  are  used,  then  they  may  be  included.  Alternatively 
goals  could  be  sunmarized  as  presented  belcw.  A  third  alternative 
would  be  to  state  the  project  level  information  in  this  section 
and  present  all  phase  (and  product)  specific  information  in  the 
next  section. 

A.  V,V&T  Requirements 
A.I.  Functional 

A. 2.  Performance 
A. 3.  Reliability 

A.  U.  Other 

B.  V,V&T  Goals  and  Measurement  Criteria  for  each  goal 

B.  I.  General 

B.2.  Product  specific 
B.3.  Phase  specific 

C.  References/ Related  Docunents 

III.  Phase  by  Phase  V,V&T  Plans 

This  section  contains  the  description  of  project  V,V&T  practices. 
Section  A  defines  the  project  approach:  phases,  their  products, 
and  the  major  reviews  and  check  points.  Where  a  practice  is 
common  to  all  phases  it  may  be  described  here.  Section  B  and 
later  sections  describe  phase  specific  activities. 

A.  Project  Background  and  Summary  Information 
A.I.  Project  Phases  and  Products 

A.  2.  Major  Reviews  (both  management  and  technical) 

B.  Requirements  Subphase  V ,V&T  Activities 

B.  I.  Summary  of  V,V&T  Goals 
B.2.  V,V&T  Activities 

B.3.  V,V&T  Techniques  and  Tools  Selected 

a.  Reviews 

b.  Methods  of  Analysis 

B.4.  Support  Requirements  and  Assunptions 
B.5.  Roles  and  Responsibilities 
B.6.  Schedules 
B.7.  Budgets 
B.8.  Personnel 

C-F.  Preliminary  Design  through  Installation  respectively  (same 
format  as  B) . 

Appendix  A     Project  and  Envirormental  Considerations 

This  section  contains  the  information  which  was  compiled  during 


St^  II. 

A.  Technical  Issues 

B.  Organizational/ Personnel  Issues 

C.  Project  Budget  and  Schedule 

D.  Computing  Resources 

Appendix  B     Technique  and  Tool  Selection  Information 
This  section  contains  the  work  sheets  of  Step  III. 

A.  Goals,  and  Related  Techniques  and  Tools 

B.  Worksheets 


Page  43 


CHAPTER  FIVE 

EXAMPLE  APPLICATIONS  OF  VALIDATION  AND  VERIFICATION  TECHNOLOGY 

This  chapter  presents  a  series  of  examples  in  which  the  concepts  of  software 
development,  software  V,V&T,  and  V,V&T  planning  are  illustrated.  The  purpose 
is  to  show  how  these  concepts  may  be  applied  in  a  variety  of  situations. 

Four  examples  are  presented,  using  an  autonobile  insurance  transaction 
processing  syst«n  as  the  system  being  developed  and  maintained.  The  first 
three  examples  describe  development  activities  in  differing  envirorments  and 
the  fourth  describes  selected  maintenance  activities. 

The  four  examples  are  structured  recognizing  that  programming  envirorments 
present  in  government  and  industry  settings  vary  significantly  in  resources 
available  to  support  V ,V&T.  The  transaction  processing  system  is  considered 
first  in  an  environment  where  only  manually  applied  techniques  are  employed. 
The  progressive  benefits  of  utilizing  automated  and  more  advanced  techniques 
can  then  be  seen  by  stuc^ying  how  V,V&T  is  done  on  the  same  application,  first 
through  adding  a  small  nunber  of  automated  tools  to  the  set  of  manual 
techniques,  and  lastly  through  use  of  a  full  complement  of  modern  automated 
tools. 

5.1  OVERVIEW  OF  EXAMPLES 

Examples  2,  3,  and  4  build  upon  Example  1.  The  tools  introduced  in  Example  2 
are  to  be  used  in  addition  to  the  manual  techniques  described  in  the  first 
example.  The  discussion  in  each  example  centers  on  the  additional 
capabilities  introduced.  Figure  5.1-1  presents  an  overview  of  the  different 
V,V&T  tools  and  techniques  which  are  used  in  the  four  examples. 

The  software  development  subphases  for  each  example  are: 

o  Requirements, 

o  Preliminary  design, 

o  Detailed  design,  and 

o  Programming  (includes  testing). 

Figure  5.1-2  shows  the  information  flow  and  relationships  among  the  four 
subphases  of  the  software  development  lifecycle  depicted  in  the  examples. 

Each  of  the  examples  will  be  presented  showing  for  each  phase: 

o  Inputs  to  the  phase, 
0  Outputs  from  the  phase, 

0  Supporting  technology  used  in  the  phase,  and 
0  Activities  which  canprise  the  phase. 
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Software  DevelopneaC  Maintenance 


Example 
(tfl) 

Manual 
Techniques 

(«)  (#3) 
Tool  Complements 

Minimal  Extensive 

(14) 

Supporting 
Technology 

Graphical  Requirements 
Represencation 

Requirements  &  Design 
Languages 

froblem  Reporting 
and  Change  Request 
Mechanisms 

StAtlc  Analysis 

Walkthroughs 
Formal  Reviews 

Cross-Referencer 
Requirements  &  Design 
Language  Analyzers 

Interface  Checker 
Dataflow  Analyzer 
Standards  Checker 

Ad  Hoc 

Dynamic  Analysis 

Functional  Testing 

Test  Coverage  Analyzers 
File  Comparator 

Assertion  Generation 
Assertion  Checking 

Formal  Analysis 

Figure  5.1-1       Overview  of  Examples 


IKTOWW.  PROSE 
W0UIR£>^>«T5 


REOUIRE>tfrrS 
DEFINITION 


OETAILED  /-i-.^ 
REQUIRCDfTS 
SPECIFICATION 


REOui  R£?crrs-aAS£i«^ ' 

fUiaiOfOi.  TEST 
CASES 


ViV 


PRELIMINART 

DESIGH 

0OCU«KT 


PRELIMINARY 

oesio* 


0ESIGJ4-8ASE3' 
FUNCTIONAL 
TEST  CASES 


OHAILEO 

DESIGN 

OOCUHENT 


OCTAILEO 
DESIGN 


3£SI5«-aAS£0  f 
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Figure  5.1-2    Example  Software  Development  Lifecycle 
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Most  activities  will  contain: 

o  V,V&T  purpose  for  the  activity, 

o  V,V&T  technique(s)  used  by  the  activity,  and 

o  Example(s). 

Tables  5.2-1,  5.3-1  f  5.4-1,  and  5.5-1  provide  a  sunmary  of  the  development  and 
V,V&T  techniques  and  activities  for  the  manual,  minimal  automation,  full 
automation,  and  maintenance  activities.  These  tables  present  a  synopsis  of 
the  examples  presented  in  this  chapter. 

The  application  area  used  in  the  examples  is  representative  of  a  large  nimber 
of  government  and  ccramercial  systans.  Transaction  processing  systems  are 
perhaps  the  most  common  of  all  commercial  systems.  Mary  banking,  billing, 
payroll,  inventory,  and  insurance  applications  are  in  this  category.  Thus, 
the  four  examples  focus  on  this  area. 

The  transaction  processing  system  is  set  in  the  context  of  an  auto  insurance 
application.  In  order  to  limit  the  size  of  the  presentations  seme 
simplifications  have  been  made  in  the  application  area.  An  expert  in  the  auto 
insurance  field  will  surely  detect  omissions  and  simplifications  in  details  of 
the  system  as  described.  The  reader  is  encouraged,  hcwever,  to  not  focus  on 
the  application  area,  but  rather  on  the  V,V&T  principles  applied.  The  details 
provided  enable  presentation  of  specific  instances  of  the  application  of  V,V&T 
techniques. 

The  Auto  Insurance  Management  System  (AIMS)  described  in  the  exairples  supports 
all  the  major  activities  of  such  a  company:  accounts  payable  (claims 
processing),  accounts  receivable  (premium  processing),  management  reports,  and 
database  management.  AIMS  must  issue  client  premium  due  notices,  checks  to 
repair  shops  (or  clients),  recommend  policies  that  should  be  cancelled, 
monitor  the  company's  day-to-day  financial  health,  and  so  forth.  Further 
details  of  the  syst«n's  requirements  are  included  in  the  first  example. 
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5.2  EXAMPLE  1:    SOFTWARE  DEVELOPMENT  USING  MANUAL  V,V&T  TECHNIQUES 

In  this  example  the  details  of  the  AIMS  are  presented  in  addition  to  the 
actual  manual  V,V&T  practices  which  are  applied  within  each  of  the  four  phases 
of  the  software  development  lifecycle. 


TABLE  5.2-1 
EXAMPLE  1  SUMMARY 
SOFIWARE  DEVELOPMENT  USING  MANUAL  V,V&T  TECHNIQUES 


SUHPIIASF.S^ 

REQUIREMENTS 

PRELIMINARY  DESIGN 

DETAILED  DESIGN 

PROGRAMMING 

INPUT 

o  Informal  prose 
Requirements 

o  Detailed  Requirements 

-  Revised  Prose 
Descr  ipt  ion 

-  Revised  Graphical 
GR  Representation 

O  V,ViT  Plan 

o  Preliminary  Design 

Ror'iimp n  t 

o  V.ViT  Plan 
o  Test  Cases 

o  Detailed  Design 

Doc  umen  t 
c  V.ViT  Plan 
o  Test  Cases 

OUTPUT 

o  Detailed  Requirements 

Specification 
o  V.ViT  Plan 
o  Initial  Test  Cases 

o  Preliminary  Design 
Document 

-  Further  Refined  GR 
System  Representation 

-  Detailed  User  Input/ 
Output  Specification 

-  Basic  Control  Flow 
Des  ign 

-  Basic  System  Infor- 
mation Specification 

o  Additional  Test  Cases 

o  Detailed  Design 

Document 
o  Additional  Test  cases 

o  System  Software 
o  Test  Results 

SUPPORTING 
TECHNOLOGY 

o  Formal  Requirements 

Rev  lows 
o  A  Graphical  Requirements 

Representation  Method 
o  Roqu I rements-based 

Functional  Testing 

o  Reviews 

o  A  Graphical  Requirements 
Representation  Method 

o  Design-based  Functional 
Testing 

o  Reviews 

o  Database  Management 

System  (DBMS) 
o  Design-based  Functional 

Test  ing 

o  Compilers 

o  Database  Management 

Sys  tem 
o  Operating  System 
o  Reviews 

ACTIVITIES 

o  Initial  Requirements 

Rev  iew 
O  Requirements  Analysis 
o  V.ViT  Planning 
o  Initial  Test  Case 

Generation 
o  Interaction  with 

cus  corner 
o  Sign-off  by  customer 

0  Refinement  of  Graphical 

Representat  ion 
o  Specify  Information 

Des  ign 

o  Design  Program  Architec- 
ure  &  Allocate  Require- 
ments 

o  Design  Basic  Control 
Flow 

o  Test  Case  Generation 
o  Preliminary  Design 
Review 

o  Detailed  Database 

Des  Ign 
o  Detailed  Module  Desgn 
o  Test  Case  Generation 
o  Critical  Design  Review 

o  Code  Development 
o  Module  Testing 
o  Function  Testing 
o  Acceptance  Twsting 
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5.2.1  Requirements  Subphase  Activity  Descriptions 

5.2.1.1.  Initial  Requirements  Review 

The  informal  prose  requirements  for  the  AIMS  is  given  in  Figure  5.2-1. 
Appropriate  management  and  technical  personnel  from  the  software  development 
group  review  these  requirements  for  completeness,  consistency,  and  correctness 
and  prepares  a  list  of  questions  which  address  particular  aspects  of  the 
requiranents.  This  list  is  then  supplied  to  the  customer  and  a  Requirements 
Review  meeting  is  scheduled  and  held  with  customer  and  user,  e.g.  clerks, 
agents.  During  the  meeting  the  questions  are  discussed  with  the  goal  being  a 
more  specific  and  unambiguous  set  of  requirements, 

V,V&T  Purpose:  To  produce  a  requirements  specification  providing  the 
foundation  from  which  more  formal  requirements  specification,  V,V&T  planning, 
and  test  planning  will  be  accomplished. 

V,V&T  Technique:  The  review  itself  is  the  V,V&T  technique  used  in  this 
activity.    Seme  of  the  questions  addressed  during  the  review  could  be: 

o  Shouldn't  a  claims  record  contain  some  kind  of  indication  as  to  the 
nature  of  the  claim?   For  example:    if  it  is  due  to  an  accident,  who 
was  at  fault? 

o  How  is  the  "reasonableness"  of  a  claim  amount  determined? 

o  How  does  one  know  what  claim  nunbers  are  valid  for  which  agents? 

0  When  is  the  pronium  rate  computed?    How  is  it  computed? 

0  Shouldn't  the  acceptance  criteria  include  provisions  for  testing  more 
than  just  the  functional  capabilities? 

5.2.1.2.  Requir«nents  Analysis 

The  requirements  analysis  involves  translation  of  the  informal  prose 
requirements  into  a  formal  representation.  This  will  result  in  the 
identification  of  other  aspects  of  the  requirements  which  will  need 
clarification  or  further  definition.  For  this  example,  the  graphical 
representation(GR)  scheme  used  is  a  modification  of  the  Systonatic  Activity 
Modeling  Method  [SAMM]. 

V,V&T  Purpose:  To  identify  inadequately  specified  requirements  such  as 
incomplete,  ambiguous,  or  otherwise  unclear  requirements  statements. 

V,V&T  Technique:  Formal  reviews  serve  as  the  V,V&T  technique  used  to  achieve 
the  above  purpose  on  this  project.  Problem  issues  which  are  identified  during 
the  requirements  analysis  are  docunented  and  distributed  to  the  customer  and  a 
second  Requironents  Review  is  sdieduled.  This  review  again  involves  dialogue 
between  the  customer  and  the  developers;  it  centers  on  the  formal 
requirements  statement  and  the  identified  issues.  The  result  is  a  revised  set 
of   requirements    in   both   formal    and   informal    forms     and     a  graphical 
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representation(GR) .  Specific  activities  which  are  performed  within  this 
review  are: 

o  Verification  that  all  requirements  have  been  correctly  represented 

using  the  formal  scheme, 
o  Identification  of  the  problems  encountered  during  the  restatement 

elaboration  of  the  requirements,  and 
o  Discussion  and  resolution  of  the  problems. 

Example: 

The  formal  representation  for  the  basic  system  and  the  accounts  payable 
function  are  shown  on  the  following  page s( figure  5.2-2).  The  graphical 
representation  is  interpreted  as  follows: 

Master  input  files  are  at  the  top  of  the  diagram  and  are  represented  by 
nunbers. 

Master  output  files  are  at  the  right  of  the  diagram  and  are  represented  by 
nunbers. 

Files  internal  to  a  data  flow  diagram  are  labeled  with  a  letter  followed  by 
a  number. 

The  upperhalf  of  figure  5.2-2  is  the  root  which  contains  five  modules,  A-E. 
The  data  flow  within  the  root  and  to  and  from  master  files  are  labeled 
according  to  their  source.  If  the  data  are  internal  to  the  root,  its 
identifier  is  preceded  by  the  module  letter. 

The  lower  half  of  figure  5.2-2  is  an  expansion  of  module  A  from  the  root. 
The  lower  left  corner  of  each  box  contains  the  parent,  i.e.  A  in  the  root. 
The  Icwer  right  corner  of  each  box  is  the  letter  designator  for  each  module, 
i.e.  A-E.  Data  created  by  the  accounts  payable  activity  is  labeled 
according  to  its  source,  e.g.  data  B.I,  a  validated  claims  transaction,  is 
created  by  module  B,  validate  claims  transaction,  and  used  by  modules  C-E. 
Data  B.2,  invalid  claims  transaction  notice,  is  created  by  B  and  put  on 
master  file  7 ,  user/client  notices. 

Some  of  the  problems  which  could  be  identified  are: 

o  What  does  the  systan  do  with  an  invalid  claims  transaction?  Solution: 

Output  a  notice  to  the  user  identifying  the  errors, 
o  The  involved  driver's  record  in  the  client's  record  needs  to  be  updated 

to  reflect  a  new  claim  due  to  an  accident.  There  does  not  appear  to  be 

enough  information  in  the  client  record  for  this.  Solution:  Add  the 

necessary  information  to  the  claims  transaction. 
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Figure  5.2-2  Requirements  Graphical  Representation 
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5.2.1.3.  V,V&T  Planning 

V,V&T  planning  is  one  aspect  of  the  overall  planning  process.  It  is 
accOTiplished  in  parallel  with  other  planning  activities  and  the  requironents 
identification  activity, 

V,V&T  Purpose:  To  choose  V,V&T  practices  which  can  be  implemented  to  suit  the 
project  needs.    The  objectives  are: 

o    Identify  the  goals  of  the  AIMS  project's  V  ,V&T  activities, 
o    Select  supporting  V ,V&T  techniques  and  tools,  and 

o  Develop  plans  for  each  phase's  V,V&T  activities.  (Plans  include  tasks, 
e.g.,  acquiring  or  developing  tools),  schedules,  responsibilities,  and 
resources. ) 

V,V&T  Technique:  The  process  described  in  Chapter  4  is  used  to  develop  the 
V,V&T  plans. 

5.2.1.4.  Initial  Test  Case  Generation 

The  AIMS  requirements  will  be  analyzed  and  test  cases  will  be  designed  to  test 
the  functional  capabilities  of  the  system.  These  test  cases  will  also  form 
the  basic  set  of  acceptance  tests. 

V,V&T  Purpose:  To  design  test  cases  which,  when  used  to  test  the  AIMS 
software,  will  maximize  the  possibility  of  revealing  the  presence  of  errors  in 
the  software. 

V,V&T  Technique:  Requirements-based  functional  testing  is  applied  to  generate 
this  initial  set  of  test  cases. 

Example: 

In  the  accounts  payable  function  a  claims  transaction  is  validated  by 
checking  (among  other  things)  that  the  claim  number  is  valid  for  the  given 
agent.  Each  agent  has  a  specified  range  in  which  claim  nunbers  associated 
with  claims  issued  by  that  agent  must  fall.  Assiming  an  agent  was  assigned 
claim  numbers  in  the  range  801000  to  801999,  test  cases  which  are  generated 
to  test  accounts  payable  should  include  claim  nunbers  as  follows. 


Test  Data  Class 

Test  Claim  Number 

ExDected  Cutout 

Comment 

Non-extremal 

801500 

None 

valid 

Non-extremal 

801317 

None 

valid 

Extremal 

801000 

None 

upper  bound 

Extremal 

801999 

None 

lower  bound 

Extronal 

800999 

Invalid  Claim  Number 

lower  bound-1 

Extremal 

802000 

Invalid  Claim  Number 

upper  bound4-1 

Special 

80 100 A 

Invalid  Claim  Number 

Special 

80100Z 

Invalid  Claim  Number 

Special 

80150 

Invalid  Claim  Number 

Special 

-01500 

Invalid  Claim  Number 

Special 

80L500 

Invalid  Claim  Number 
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5.2.2  Preliminary  Design  Subpiiase  Activitj^  Descriptions 

5.2.2.1.  Refinement  of  Graphical  Representation 

The  GR  diagrams  developed  during  requirements  analysis  will  be  deconposed  to 
reflect  l^e  requirements  for  the  system  in  more  detail. 

V,V&T  Purpose;  The  conpleteness  and  consistency  of  the  GR  description  of  the 
requironents  and  preliminary  design  should  be  ensured, 

V,V&T  Technique:    A  review  of  the  resulting   diagrams  will   be   performed  to 

verify: 

o  that  all  of  the  basic  activities  necessary  to  perform  a  particular  function 
have  been  identified, 

o  that  all  inputs  and  outputs  required  by  each  activity  have  been  identified, 
and 

o  the  consistency  and  conpleteness  of  the  data  flows. 
Example: 

Within  the  accounts  payable  function  there  is  no  indication  as  to  the  action 
which  is  to  be  taken  when  a  claim  transaction  is  processed  for  a  claim  which 
has  been  previously  entered.  This  error  would  be  discovered  during  the 
review  of  the  GR  activity  for  the  accounts  payable  function. 

5.2.2.2.  Specify  Information  Design 

The  preliminary  design  of  the  information  consists  of  a  detailed  user 
input/ output  specification  and  a  description  of  the  basic  content  and 
structure  of  the  data  used  by  the  systen.  The  detailed  user  input/output 
specification  essentially  amounts  to  preparing  a  user's  manual  for  the  syston. 
The  formats  used  to  input  claims  and  pronium  payment  transactions  are  defined 
as  well  as  the  output  responses.  The  printed  report  formats  for  the 
management  reports  are  also  defined.  Specification  of  the  basic  data 
structures  and  content  will  consist  of  identification  of  variables  and  records 
needed       the  system,  and  the  relationships  which  exist  among  them. 

V,V&T  Purpose:  The  V^V&T  purpose  in  this  activity  is  twofold.  First,  the 
detailed  user  specifications  need  to  be  shown  to  be  usable  and  that  they 
satisfy  the  needs  of  the  user.  Second,  the  systan  data  structures  and  content 
need  to  be  verified  and  shown  to  be  conplete  (i.e.,  that  which  is  required  to 
perform  all  system  functions)  and  correct  (i.e.,  the  data  types  and 
relation^ips  are  consistent  with  the  functions  which  need  to  be  performed). 

V,V&T  Technique: 

0  A  formal  review  will  be  held  with  the  custaner  to  review  the  detailed  user 
input/output  specifications.    Ihis  will  be  preceded  by  informal  dialogue 
between  the  user  conmunity  and  the  developers  to  aid  in  the  development 
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of  the  specifications.    Once  satisfied,  the  custcmer  will  formally  sign-off 
on  the  specification, 
o  Formal  inspections  of  the  system  data  structures  and  content  will  be 
performed. 

Example: 

Discovered  by  the  custcmer  participating  in  the  formal  review  of  the 
detailed  input/ output  spec  was  that  a  client  is  not  always  the  owner  of  the 
car,  so  that  lien-holder  information  needs  to  be  included  in  the  client 
record. 

5.2.2.3.  Design  Program  Architecture  &  Allocate  Requirements 

The  progran  architecture  design  gives  a  complete  high-level  description  of  the 
software.  It  refines  and  groups  functions  defining  software  components  and 
interfaces. 

V,V&T  Purpose:  Requirements  are  cross-rel'erenced  by  the  design  to  ensure  that 
all  the  requirements  have  been  addressed. 

V,V&T  Technique:    Specification  tracing. 

Example: 

A  ccraplete  set  of  cross-references  is  defined  and  maintained.  These  show 
the  evolution  from  the  prose  requirements  to  the  requirements  represented  by 
the  GR  and  finally  to  the  components  identified  in  the  design. 

5.2.2.4.  Design  Basic  Control  Flow 

The  GR  represents  the  data  flow  within  a  system  but  only  shows  control  flew  in 
an  implicit  way.  The  sys ton's  control  flow  therefore  needs  to  be  explicitly 
designed.  The  activities  identified  in  the  GR  will  need  to  be  mapped  into 
modules.  The  control  flew  between  modules  must  also  be  described.  This  is 
done  using  an  informal  design  language.  This  defines  the  program 
architecture.  The  hierarchical  structure  of  the  modules  comprising  the  system 
are  developed. 

V,V&T  Purpose:  To  produce  a  correct  and  understandable  description  of  the 
basic  control  flow  of  the  syst«n. 

V,V&T  Technique:  An  inspection  of  the  control  flow  design  will  be  performed 
to  verify: 

o  its  consistency  with  the  GR  representation, 
o  correctness  of  the  high  level  logic,  and 

o  the  quality  of  the  modularization,  i.e.,  are  the  functional  boundaries 
natural? 
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5.2.2.5.  Test  Case  Generation 

V,V&T  Purpose:  Generate  test  data  that  will  exercise  and  test  each  function, 
and  also  demonstrate  the  code  is  consistent  with  the  design, 

V,V&T  Technique:    Desigivbased  functional  testing. 

Example: 

Test  cases  for  a  function  which  adds  the  amount  of  the  premium  payment  to 
the  payout  account  would  include:  a  negative  (or  zero)  amount,  an  amount 
which  is  greater  than  zero  but  less  than  that  which  would  leave  the  balance 
larger  than  the  maximum  allowed,  and  one  which  would  leave  the  balance 
greater  than  the  maximum  allowed. 

5.2.2.6.  Preliminary  Design  Revia^ 

At  the  ccmpletion  of  the  preliminary  design  activity,  a  formal  review  is  held. 
This  involves  management  and  technical  staff  representing  the  developer  and 
the  custaner/user.  It  covers  all  aspects  of  the  design.  Results  of  V,V&T 
activities  are  revi»/ed.  Management  of  custaner/user  and  developer  sign-off 
of  acceptance  is  required. 

5.2.3  Detailed  Design  Sublease  Activity  Descriptions 
5.2.3.1.    Detailed  Database  Design 

The  format  and  structure  of  the  data  to  be  stored  in  the  syst^  database  is 
designed.  This  includes  description  of  data  which  are  logically  related  in 
the  form  of  records,  and  the  relationships  which  exist  between  records.  The 
logical  structure  of  the  database  will  be  described  using  a  grap*iical  database 
design  representation.  Record  descriptions  will  be  specified  in  a  Data 
Definition  Language.    Examples  are  shown  in  figures  5.2-3  and  5.2-4. 

In  figure  5.2-3,  ovals  represent  record  access  (key)  fields,  boxes  represent 
records,  "1:M"  means  that  for  each  client  record  there  are  potentially  many  (1 
or  more)  claims  records. 


Figure  5»2-3  Sample  Database  Schema  Showing  Client^Claims  Relation 
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record  name  is  CLAIMS 


location  mode  is  calc  in  CALC-KEY  using  POLICY-NUM 


01 

CLAIM-NUM 
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9 
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9 
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01 

DRIVER 

02  LAST 
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X 

(15) 

02  FIRST 

PIC 

X 

(15) 

02  MIDDLE-INTL 

PIC 

X 

01 

PAYEE 

02  NAME 
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X 
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03  STREET 
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X 
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03  CITY 
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X 
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03  STATE 

PIC 

X 
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03  ZIP 
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X 

(5) 

01 

POLICY-NUM 

PIC 

9 

(8) 

01 

AGENT 

PIC 

9 

(5) 

Figure  5.2-4  Sarrple  CLAIMS  Record  Description 

V,V&T  Purpose:  The  database  design  needs  to  be  verified  for  consistency  with 
the  preliminary  design.  Moreover,  the  database  structure  will  be  verified  to 
ensure  that  it  is  correct  and  is  reasonable  with  respect  to  potential  storage 
consunption  and  access  time. 

V,V&T  Technique:  An  inspection  of  the  database  design  is  performed  to  ensure 
that  Uie  above  V,V&T  purpose  is  met. 

Example: 

During  the  inspection  of  the  database  design  an  error  is  found  in  the  claims 
record  (Figure  5.2-4)  where  POLICY-NUM  is  identified  as  the  key  field  where 
as  the  schema  diagram  (Figure  5.2-3)  indicates  CLAIM-NUM.  The  solution  is 
to  change  the  key  field  in  the  claims  record  description  to  CLAIM-NUM. 

5.2.3.2.    Detailed  Module  Design 

Detailed  module  design  includes  for  each  module  a  description  of  the  function 
performed  and  descriptions  of  input  and  output  data,  as  well  as  a  high  level 
description  of  how  the  function  is  to  be  done  (i.e.,  the  algorithm  used). 

V,V&T  Purpose:  To  show  that  1.  all  of  the  system's  functional  capabilities 
are  addressed  by  one  or  more  modules  and  2.  each  module  addresses  one  or  more 
syston  functions.  Moreover,  relation^ips  among  and  interfaces  between  all 
modules  are  identified  and  verified. 

V,V&T  Technique: 

0  Inspections  of  the  syston  modules  are  held.    Activities  performed  during 
the  inspections  include  manual  checking  of  the  module  interfaces  to 
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ensure  that  all  modules  are  used  and  that  their'  inputs  and  outputs  are 
consistent.  Inspections  are  also  used  to  verify  informally  the  correctness 
of  the  algorithms  used. 

o  Requironents  tracing  is  accomplished  by  identifying  each  module  with 
the  lowest  level  GR  activity  (from  the  preliminary  design)  in  which  the 
module  is  contained. 

Example: 

A  module  which  updates  the  date  and  time  of  the  last  access  to  the  payout 
account  record  has  as  one  of  its  inputs  the  premiun  payment  transaction. 
However,  manual  interface  checking  detects  an  inconsistency  whereby  the 
pronium  payment  transaction  is  not  supplied.  As  it  turns  out,  the 
transaction  is  not  used  within  the  module  and  is  deleted  as  an  input. 

5.2.3.3*    Test  Case  Generation 

This  involves  refining  and  adding  to  test  data  previously  developed. 

V,V&T  Purpose:  Test  cases  are  developed  to  exercise  and  test  the  internal 
structures  and  functions  of  modules. 

V,V&T  Technique: 

o  Branch  testing 
0  Path  testing 

Example: 

The  module  which  validates  a  claim  number  checks  for  six    error  conditions. 
Associated   with  these  conditions  are  three  actions.    Test  data  is  developed 
to  exercise  all  combinations  of  error  conditions  and  resulting  actions,  that 
is  all  branches  and  all  paths  throu^  the  modules, 

5.2.3.4.    Critical  Design  Review  (CDR) 

At  the  conpletion  of  the  detailed  design  a  formal  detailed  design  review  is 
held.  This  primarily  involves  project  management  and  technical  personnel  and 
covers  all  aspects  of  the  design  (including  the  test  cases).  Management 
sign-off  of  their  acceptance  of  the  design  is  required. 

5.2.4  Programming  Subphase  Activity  Descriptions 

5.2.4,1,    Code  Development 

The  detailed  design  of  a  given  component  provides  the  information  needed  to 
write  the  code  for  that  ccmponent  in  the  host  programming  language  ,e.g. , 
COBOL.  Once  written,  the  code  is  entered  into  the  ccxnputer  and  all 
conpilation  errors  are  ronoved. 
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V,V&T  Purpose:    V,V&T  of  the  compiled  code  is  performed  to: 

o  Verify  the  consistency  of  the  code  with  the  detailed  design, 
o  Identify  errors,  and 

o  Ensure  adherence  to  programming  standards. 
V,V&T  Technique:    Formal  reviavs  of  each  system  module. 
Example: 

During  an  inspection  of  "issue  policy  notices"  module  the  section  of  code 
which  is  responsible  for  issuing  a  premium  due  notice  is  found  to  be  in 
error.  The  error  is  that  the  premiun  due  notice  is  printed  without  having 
the  appropriate  data  moved  into  the  printer  buffer.  A  sample  portion  of  the 
inspection  checklist  used  is  shown  below  in  Figure  5.2-5.  This  particular 
error  is  discovered  using  question  two  under  "data  reference. " 


DATA  DECLARATION 

1 .  Are  all  variables  declared? 

2.  Are  the  correct  attributes  assigned? 

3.  Are  variables  properly  initialized? 

4.  Are  variable  naming  conventions  followed? 

5.  Is  the  proper  explanatory  ccmment  included  for  each  variable? 

DATA  REFERENCE 

1 .  Are  there  any  unreferenced  variables? 

2.  Are  there  any  references  to  unassigned  variables? 

3.  Are  subscripts  within  range? 

4.  Are  there  off-by-one  errors  in  subscript  canputations? 


Figure  5.2-5     Sample  Portion  of  Code  Inspection  Checklist 
5.2.4.2.    Module  Testing 

An  incremental,  bottorrnup  testing  strategy  is  used  to  test  the  AIMS  modules. 
This  involves  individually  testing  the  lowest  level  modules;  then  combining 
and  testing  those  modules  with  the  higher  level  modules  which  call  them.  The 
process  continues  until  all  modules  are  canbined  into  the  complete  systan. 
Test  drivers  are  written  to  control  the  testing  of  the  individual  modules. 
The  test  data  used  is  that  created  by  design-based  functional  testing  which 
were  generated  from  analyses  of  the  functional,  structural  and  interface 
specifications  of  the  individual  modules  during  detailed  design. 

V,V&T  Purpose:    To  reveal  errors  present  in  the  individual  modules. 
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5.2.4.3.  Function  Testing 

Function  testing  of  AIMS  uses  the  test  cases  developed  frcm  requirements- based 
functional  testing  during  preliminary  design  to  test  the  functional 
capabilities  of  the  AIMS  software. 

V,V&T  Purpose:  To  reveal  errors  where  the  software  fails  to  perform  a 
function  as  specified  in  the  requirements. 

5.2.4.4.  Acceptance  Testing 

Acceptance  testing  is  similar  to  function  testing  in  that  it  consists  of  a 
subset  of  the  same  test  cases.  But  where  the  purpose  of  function  testing  is 
to  reveal  the  presence  of  errors,  acceptance  testing  is  to  demonstrate  that 
the  software  performs  according  to  its  specification.  Acceptance  testing  is  a 
formal  procedure  and  requires  custcnier  sign-off. 

5.3  EXAMPLE  2:  SOFTWARE  DEVELOPMENT  USING  A  MINIMALLY  AUTOMATED  V,V&T  TOOL 
SET 

Ihe  minimally  automated  V,V&T  tool  set  consists  of  a  set  of  commonly  available 
tools.  These  are  used  to  supplement  the  manual  techniques  described  earlier. 
The  tools  contained  in  the  minimal  set  and  the  lifecycle  phase  in  which  they 
are  applied  are  shown  below. 


TABLE  5.3-1 
EXAMPLE  2  SUMMARY 
SOFTWARE  DEVELOPMENT  USING  A  MINIMALLY  AUTOMATED  V,V&T  TOOL  SET 


SUBPMSES. 

Rfouinocr? 

ptaiWRAin  oaia 

DETAani  otsia 

:i»uT 

•    (No  A4dU1on«l  Input) 

So»Clf1Clt1on  InclualBfl 
Ul«  o'  flKulrCMntl 
Sp*c1f1Cit1on  L«n9u49« 

•    Pr«M»1r\«ry  OoilTn 

Doci«*nt  Inc1ud1n9  UM 
of  IUou1rw»nt  irtd 
D«il5ti  Se«c1f Icitlon 

»    0«t>11*4  0Ml9n 

QoctMnt  Includlif  I.) 
of  0«l9n  Specification 

OUTPUT 

$p*c1f(:itlon  Including 
Ut*  of  EL*Qu1rw»nu 
Sp«c1f1ut1on  U»9u«9* 

•    Pr»11»1ntr)r  0*«l9n 

Ooo«nt  Including 
Furtfl.r  Ji*  of  9»<jutr»- 
mnu  Sp«<:1ff cation 
On  1  <p\  Lin9vM9* 

OociMnt  Including  Us* 
of  tnutrtrmnt  »na 
D*ii9n  So«cU1ut1on 

•    (Xo  Dtw  OutpuU ) 

SUPPORTING 
TtCHNOLOSr 

•  R»ou1i  mawu  SMC^fl  ca- 
tion Lin<;u49* 

•  Btqutnwnu  Sp«c1'U«- 
tion  Ltn9u<9t  •Ji«lyztr 

•  l(»<rjir««»»otJ  Sp«cU1c»- 
tlon  linqu49« 

*  TooUSuPDorTM  Ocilfn 

•    0»i1?n  So«c1f1cition 

*  Croil-*tf  •'"•nctr 

*  Ttst  COY»r»g»  Analyitr 

*  F11«  ConMritor 

ACTIVITIES 

•   flTCufrcMntx  Analyiis 

•  So«1fy  IntorBttlon 
Otilqn 

•  Oeiign  t«s1c  Control 
Hon 

•    [>*ui1«4  Hodul* 
Ottlgn 

•  Cod*  0«v«1oi>»flt 

•  noouli  T«ttln9 

•  Function  Ttitlnf 

o  Requirements 
-  Requirements  representation  language  analyzer 

o  Preliminary  and  Detailed  Design 
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-  Design  representation  language  analyzer 
o  Code 

-  Cross  reference  generator 

-  File  comparator 

-  Test  coverage  analyzer 

5.3.1  Requirements  Subphase  Activity  Description 

A  tool- supported  requirements  statement  language  is  used  to  formally  specify 
the  requirements.  It  is  used  in  conjunction  with  the  graphical  represention 
technique  which  analyzes  the  gross  requirements  while  the  language 
representation  is  used  to  perform  a  more  detailed  analysis.  It  is  used  to 
directly  support  the  V,V&T  purpose  to  identify  inadequately  specified 
requirements  in  that  a  consistency  analyzer  is  part  of  the  tool.  The  reports 
generated  by  the  tool  are  included  with  the  other  material  for  requirements 
reviews. 

Example: 

A  Problem  Statement  Language  (PSL)  [TEIC]  description  of  the  accounts 
payable  function  is  shown  in  Figure  5.3-1. 


/*  Auto  Insurance  Management  Systems  (AIMS) 
Accounts  payable  •/ 

INPUT:  clairas-transaction-file; 

OUTPUT:  claim-payment>check,  client-notices; 

SET:    aims-data- base 

INTERFACE:  client; 

GENERATES:  claims-transactions; 

RECEIVES:    claim- payment-checks; 
PROCESS:    accounts- pay able; 

UPDATES:  aims-data-base; 

RECEIVES:  claims-transactions; 

GENERATES:  claim-payment-checks; 
EOF 


Figure  5.3-1     PSL  Specification  for  Accounts  Payable 

In  this  example,  the  Problem  Statement  Analyzer  (PSA) [TEIC]  would  identify 
an  inconsistency  with  respect  to  the  accounts  payable  process.  Accounts 
payable  generates  clairo-payment-checks  which  have  not  been  previously 
defined  (it  is  misspelled  in  the  OUTPUT  definition). 

5.3.2  Preliminary  Design  Subphase  Activity  Descriptions 

5.3.2.1.    Specify  Information  Design 

A  formal  requirements  specification  language  (such  as  PSL)  can  be  used  to 
specify  a  high-level  system  information  design.  Reports  produced  by  the  tool 
are  included  in  the  preliminary  design  document  and  are   analyzed   during  the 
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reviews, 

5.3.2,2.    Design  Basic  Control  Flow 

One  of  t^e  difficulties  encountered  in  the  use  of  an  informal  design  language 
is  that  of  ensuring  a  consistent  representation  for  all  modules.  The  use  of  a 
tool-supported  design  language,  such  as  Program  Design  Language  (PDL)  [CAIN], 
prevents  this  frcHii  happening. 

Example: 

Figure  5.3-2  shews  the  PDL  description  for  "Issue  Policy  Notices".  The  PEL 
for  all  modules  is  included  in  the  reviews  and  walk-throughs. 


Issue  Policy  Notices 
Initialize 

Fetch  first  client  record 
do  until  end-of-file  on  client  file 
if  premiun  is  due 

Issue  premium  due  notice 
else  if  premiun  past  due 

Issue  cancellation  notice 
endif 

Fetch  next  client  record 

enddo 


Figure  5.3-2     Preliminary  Design  PDL  of  "Issue  Policy  Notices"  Function 
5.3.3  Detailed  Design  Subphase  Activity  Description 

As  in  the  preliminary  design  described  above,  a  formal,  tool-supported  design 
language  is  used  to  describe  modules  and  their  algorithms.  The  reports 
generated  by  the  tool  are  included  with  the  information  used  in  the  reviews 
and  walkthroughs. 

During  the  inspection  of  this  module  two  problems,  an  error  and  a  standards 
violation,  are  identified.  The  error  is  that  the  program  may  very  likely 
issue  more  than  one  premitm  notice.  The  problem  occurs  in  line  six.  If 
currents-date  is  greater  than  or  equal  to  pronium-due-date  -  40  then 
current-date  +1  (when  the  program  is  next  executed)  will  also  be  greater.  One 
solution  is  to  introduce  an  additional  boolean  variable  in  the  client  record, 
"issued",  which  is  initially  false  but  set  to  true  when  the  notice  is  issued. 
Line  six  then  becomes: 

if  (current-date  >=   premium-due-date  -  ^0)  and  not  issued 

The  standards  violation  is  a  result  of  the  use  of  the  integer  literal  constant 
"40"  in  line  six,  particularly  since  this  value  m^  be  subject  to  change,  A 
program  constant  called  "response- interval"  should  be  defined  with  the  value 
40  and  referenced  in  line  six. 
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Example: 

Figure  5.3-3  shews  the  PDL  of  the  detailed  design  for  "Issue  Policy  Notices" 
from  the  preliminary  design  shown  in  Figure  5.3-2. 

Issue  Policy  Notices 


1  Get  current-date 

2  Issue  operator  instructions 

3  Open  data  base 

4  Fetch  first  client  record 

5  do.  until  end-of-file  on  client  file 

6  if  current-date  >  =    premiun-due-date  -  40 

7  Issue  premium  notice 

8  else  if  current^date  i  premium-due-date 

9  Issue  cancellation  notice 

10  endif 

11  Fetch  next  client  record 

12  enddo 

13  Close  database 


Issue  operator  instructions 

14  Display  program  start  message  and  date 

15  Display  prepare  printer  message 

16  Wait  until  printer  ready 


Figure  5.3-3    Detailed  Design  PDL  of  "Issue  Policy  Notices"  function 
5.3.4  Progranming  Subphase  Activity  Description 

5.3.4.1.  Code  Development 

In  addition  to  that  described  earlier,  a  cross-ref erencer  is  used  to  produce 
cross-reference  lists  of  all  identifiers  used  by  a  program.  This  list  is 
included  with  the  source  code  listings  for  module  inspections. 

Example: 

A  careful  examination  of  the  cross-reference  listing  of   module  ISSUE-CHECK 
in  ACCOUNTS- PAYABLE  during  the  code  inspection  indicated  that  two  variables, 
PAYOUT- ACCOUNT-BAL  and  PAYOUT-ACCT-BAL,  were  referenced.    The  error  was  that 
P AYOUT- ACCOUNT- BAL,  should  have  been  coded  "PAYOUT-ACCT-BAL." 

5.3.4.2.  Module  Testing 

A  test  coverage  analyzer  is  used  to  supplement  module  testing  as  described  in 
Table  5.3-1.  Each  module  to  be  tested  is  instnmented  to  collect  execution 
frequency  counts  and  then  executed.  The  execution  counts  for  each  statement 
are  then  listed  with  the  corresponding  statement  by  a  post-execution  routine. 
Untested  or  poorly  tested    portions   of    the   module   can    be    identified  and 
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additional  test  cases  can  be  generated  to  test  those  specific  segments. 
Example: 

ACCOUNTS- PAYABLE  processes  claims  transactions  read  from  a  file  which 
contains  a  given  day's  claims.  The  module  contains  a  check  to  verify  that 
each  record  is  indeed  a  claims  transaction  and,  if  not,  invokes  an  error 
handling  routine  which  logs  the  error.  Use  of  a  test  coverage  analyzer 
showed  that  this  particular  situation  did  not  arise  during  testing  of  the 
module  using  the  tests  created  during  detailed  design.  As  a  result,  those 
tests  are  supplemented  with  invalid  claims  transactions  and  the  module 
retested.  This,  in  turn,  results  in  an  error  being  revealed  whereby  the 
error  handler  responds  with  an  incorrect  output  response. 

5.3.^.3.    Function  Testing 

Function  testing  is  supplemented  with  the  use  of  a  file  conparator. 
Associated  with  each  of  the  requirements-based  functional  test  cases  is  the 
expected  output.  ITiis  is  stored  on  a  file  in  the  exact  format  expected  to  be 
produced.  the  AIMS  software  is  tested,  the  resulting  output  is  stored  on 

a  separate  file,  A  file  comparator  is  used  to  detect  automatically  any 
discrepancies  which  may  have  occurred. 

Example: 

In  preparing  the  test  cases  for  the  New  Clients  report,  a  form  is  used  which 
formats  the  expected  output  data  in  accordance  with  the  specification.  Each 
report  corresponding  to  a  given  test  case  is  then  stored  on  a  file  in  the 
order  in  which  the  tests  are  to  be  executed.  Testing  is  then  performed  and 
the  actual  output  is  compared  to  the  expected  output  using  a  file 
comparator.  The  results  show  the  presence  of  two  errors,  a  format  error  and 
a  data  output  error.  The  format  error  is  a  misalignment  caused  by  incorrect 
spacing  between  output  fields.  The  data  output  error  is  a  missing  agent 
name  which  is  to  be  printed  with  the  agent  nimber. 
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5.4  EXAMPLE  3:    SCFIVARE  DEVELOPMENT  USING  A  FULLY  AUTOMATED  V,V&T  TOOL  SET 

The  fully  autanated  V,V&T  tool  set  is  comprised  of  commonly  used  tools  and 
techniques,  and  includes  those  tools  contained  in  the  minimal  tool  set 
described  earlier  as  well  as  those  described  in  this  section.  The  additional 
tools  and  the  applicable  lifecycle  phase  are  shown  below. 

0    Preliminary  Design 

-  Assertion  generation 

o    Detailed  Design 

-  Assertion  generation 

o  Code 

-  Interface  checker 

-  Data  flew  analyzer 

-  Assertion  processor 

-  Standards  analyzer 

TABLE  5.4-1 
EXAMPLE  3  SUMMARY 
SOFTWARE  DEVELOPMENT  USING  A  FULLY  AUTOMATED  V,V&T  TOOL  SET 


mLiniMRT  oQia      I      oaAiLED  oesisi 

•    (l«e  MdttlORt  to 
mnlHl  Tool  S«t) 

•    (He  Addltloiul  Inputs) 

OocvMfit  Includlnf 
Ati«rc1oni 

AsMrclani 

OUTW 

•    [M  AddUlont  to 
mnliBl  Tool  S«t) 

OooMnt  Inclvidinf 
Aiitrtlons  loout  txt 

•    0«t<11t4  Otltp 

AedltlOTMl  A(l«rC1«« 

•   (He  Mdltloiiil  OutBuu) 

TicmoujCT 

•   (Me  Mtflttont  ta 
mnlMl  Tool  S*t) 

•   Asi«rt1e«  Stnaratlon 

t   Asttrtlon  fi*fl«rit1on 

•  [nt*r'tet  Chtcktr 

•  04U  Flow  Antlyztr 

•  illtrtlsn  'njctjior 

AcntmEs 

•    (He  Addltlani  to 
mnlMl  Tsol  Sat) 

•  Oeslftt  ittic  Central 
nam 

Onl9n 

»    Com  9tT«lcon>nt 

5.4.1  Requirements  Subphase  Activity  Description 
(No  additions  to  minimal  tool  set.) 

5.4.2  Preliminary  Design  Subphase  Activity  Description 

V,V&T  Technique:  Assertion  generation  is  used  to  specify  the  desired 
functional  properties  of  the  individual  modules.  This  is  done,  by  including  in 
the  module  specifications  input  and,  to  the  extent  possible,  output 
assertions. 
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Example: 

Policy  numbers  are  stored  in  the  database  in  blocks  of  arrays  where  each 
block  contains  a  fixed  nunber  (n)  of  policy  nunbers  (policy-nun)  and  the 
address  (policy-addr)  of  their  associated  client  records.  Policy  nunbers 
are  stored  in  the  policy-nun  array  in  ascending  order.  A  procedure, 
find-policy,  is  called  to  search  the  policy-nun  array  for  a  supplied  policy 
nunber  and  return  the  address  of  its  client  record.  If  the  supplied  policy 
nunber  is  not  found  an  address  of  zero  is  returned.  The  input  and  output 
assertions  which  capture  the  functional  properties  of  find-policy  are  given 
below. 

1)  /*  assert  input  poli^r-nun  (IX  =  nun<  =  policy-nun  (m)  */  and 

2)  /*  assert  input  forall  i  in.  1 . . .n-1  :polcy-nun  (i)<  =  policy-nun  (i+1)  V 

3)  /*  assert  output  (exists  in  i  in  1 .  ..n  :  nun  =  police-nun  (i))  */  or. 

4)  /*  assert  output  (addr=0  and  forall  i  in.  1 . .  .n:nun  =  policy-nun  (i) )  */ 

5.4.3  Detailed  Design  Sub^iase  Activity  Description 

V,V&T  Technique:  Assertions  are  generated  to  include  algorithmic  detail  in 
addition  to  input  and  output  specifications  of  the  functional  properties  of 
the  individual  modules. 

Example: 

The  example  in  the  previous  section  describes  the  find-policy  procedure  and 
specifies  the  input  and  output  assertions  associated  with  it.  Shewn  in 
figure  5.4-1  is  the  PDL  for  find-policy  which  is  implemented  using  a  binary 
search  algorithm. 

Ihe  input  and  output  assertions  capture  the  functional  properties  of  the 
procedure  independent  of  the  algorithm  used  to  implement  the  search. 
Assertions  1,  2  and  3,  however,  capture  conditions  which  are  very  dependent 
upon  the  algorithm.  Assertion  1  is  always  correct  whenever  nun  is  in  the 
policy-nun  array.  If  nun  is  not  in  the  array,  assertion  1  is  violated  the 
last  time  ttirough  the  loop  (when  high  =  low).  This  is  an  acceptable  result, 
however,  in  that  nun  should  be  a  valid  policy  nunber. 

find-policy: 

/*  searches  sorted  global  array  policy-nun  for  nun  (input  argunent)  and,  if 
found,  returns  the  associated  policy-addr  in  addr  (output  argunent).  If 
not  found  a  zero  is  returned  in  addr  V 

/*  assert  input  policy-nun  (IX  =  nun<  =  policy-nun  (n)  */ 

/•  assert  input  forall  i  in  1 .  ..n-1 :  policy-nun  (i)<  =  policy-nun  (i+1)  */ 

set  addr  to  0 
set  lew  tQ_  1 
set  high  to  n 
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do  until  high<  low  or  nun  =  policy-nun  (i) 
(1)  /*  assert  1<  =  low<  r  high<  =  n  and  policy-nun  (lowX  =  nun<= 
policy-nun  (high)  */ 
set  m  to  (low  +  high)  /2 
if  nun<  policy-nun  (i) 

else  if  nun>  Dolicv-nun  (i) 

else  goto  successful 
enddo 

/*  unsuccessful  */ 

(2)  /*  assert  high  =  low-1  and  policy-nun  (highX  nun<  policy-nun  (low)  •/ 
/*  assert  output  addr  =  0  gnd  for  all  i  in  1 ,  ..n:  nun  4  policy-nun  (i)  «/ 
return 

/*successful*/ 

set  addr  to  policy-nun  (i) 
(3)  /*  assert  1<  =  lcw<  =  m<  =  hiRh<  =  n  and  nun  =  policy-nun  (m) 
/*  assert  cutout  exists  i  in  1...n:  nun  =  policy-nun  (i)  */ 
return 

»/ 

end  find-policy; 

Figure  5.4-1    Detailed  PDL  with  ASSERTIONS 

5.4.4  Programming  Subphase  Activity  Descriptions 

5.4.4.1.    Code  Development 

The  code  development  activities  described  in  earlier  sections  are  supplanented 
in  a  full  tool  set  envirorment  with  an  interface  checker,  data  flow  analyzer 
and  standards  analyzer.  These  tools  can  be  separate  but  are  often  included  as 
capabilities  provided  by  a  single  tool.  They  are  all  static  analysis 
techniques  and  are  therefore  applied  prior  to  software  testing.  The  output 
which  results  from  each  of  the  capabilities  is  included  with  the  material  for 
the  formal  code  inspections. 

V,V&T  Techniques: 

0  Interface  checking  is  used  to  check  the  consistency  of  the  interfaces 
between  modules. 

Example: 

An  error  is  detected  between  the  module  which  reads  client  records  for 
premium  payment  processing  and  the  "find-policy"  module.  It  is  an 
inconsistency  in  the  type  of  the  argunents  for  the  policy  nunbers. 
"Find-policy"    is    being  called  with  a  policy  nunber  of  type  character  where 
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it  should  be  type  integer, 

o  Data  flow  analysis  is  used  to  identify  variable  reference/definition 
ancmalies. 

Example: 

When  data  flow  analysis  is  performed  on  the  module  which  updates  the  payout 
account  with  a  premiim  payment,  a  reference  to  an  uninitialized  variable  is 
noted.  The  variable  should  contain  the  current  date  and  time  and  is  used  to 
update  the  date  and  time  of  the  last  change  to  the  payout  account,  A  call 
to  the  routine  which  updates  the  time  and  date  should  be  made  prior  to  the 
reference, 

o  Standards'  analyzers  are  used  to  assure  adherence  to  program  coding  and 
docunentation  standards.    One  of  the  primary  capabilities  provided  by  most 
commonly  available  standards*  analyzers  is  the  notification  of  the  use  of 
non-standard  language  features. 

Example: 

One  of  the  requirements  for  the  AIMS  software  is  that  it  be  portable.  To 
assist  in  the  development  of  portable  code,  a  COBOL  standards'  analyzer  is 
used.  All  places  where  a  standards'  violation  occurs  is  either  changed  or 
justified.  Even  trivial  nonstandard  features  such  as  the  use  of  the 
abbreviation  "DISP"  for  "DISPLAY"  are  detected.  In  addition,  a  variety  of 
undesirable  standard  language  constructs  such  as  the  "ALTER"  statonent  and 
"NEXT  SENTENCE"  clause  are  detected  with  the  tool. 

5.4.4,2.    Module  Testing 

The  module  testing  activities  described  in  earlier  sections  are  supplemented 
in  a  full  tool  set  envirorment  witti  a  dynamic  assertions  processor.  This 
processor  is  generally  included  as  part  of  a  broader  dynamic  analysis  tool 
which  includes,  for  example,  statement  execution  counts. 

V,V&T  Technique:  Assertions  processor:  A  dynamic  assertions  processor 
translates  assertions,  usually  specified  as  part  of  the  source  program,  into 
source  language  stat«nents  which  check  the  validity  of  the  assertion  during 
program  execution.  Generally,  when  an  assertion  is  violated,  an  informative 
message  is  output. 

Example: 

Figure  5.4-2  shows  a  portion  of  a  FORTRAN  implementation  of  the  find-policy 
routine  from  Figure  5,4-1,  Also  shown  is  an  example  of  an  assertion 
violation  message  which  was  printed  when  the  assertion  in  line  14  of  the 
program  was  violated  (i,e, ,  false)  during  program  execution.  Subsequent 
analysis  of  the  problem  indicated  that  the  error  was  an  incorrect  coding  of 
line  18  from  the  PDL  where  HIGH  should  have  been  set  to  M  -  1,  not  M  +  1. 
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13  100  CONTINUE 

14  C»  ASSERTd.LE.LOW.AND.LOW.LE.HIGH.AND.HIGH.LE.N 

15  C«  . AND. POLNUMC LOW). LE. NUM. AND. NUM. LE.POLNUMC HIGH)) 

16  M  =  (LOW  +  HIGH)/2 

17  IF  (  NUM  .LT.  POLNUMC  M)  )  THEN 

18  HIGH  =  M  +  1 

19  ELSE  IF  (  NUM  .GT.  POLNUMC M)  )  THEN 

20  LOW  =  M  +  1 

21  ELSE 

22  GO  TO  200 

23  ENDIF 

24  IFC HIGH. LE. LOW. AND. NUM. NE. POLNUMC M))  GO  TO  100 


»»«  ASSERTION  VIOLATION  AT  LINE  1  4  OF  SUBROUTINE  FNDPOL: 
CURRENT  EXECUTION  COUNT  =  2 
LOW  =  1,  HIGH  =  65,  N  =  64,  NUM  =  22707, 
POLNUMC  LOW)  r  16747,  POLNUMC  HIGH)  =  36757 


Figure  5.4-2 

Find-policy  Subroutine  and  Corresponding  Assertion  Violation  Message 
5.5  EXAMPLE  4:    SOFTVARE  MAINTENANCE 

System  maintenance  activities  involve  the  processing  of  changes  to  the  syston 
software  and  docunentation  necessary  to  correct  an  error  or  to  enhance  or 
alter  the  syston's  functional  capabilities.  The  maintenance  activities 
described  in  this  example  are  supported  by  the  fully  automated  V,V&T  tool  set, 
described  in  earlier  examples,  augmented  with  software  configuration  control 
procedures. 

Maintenance  is  generally  divided  into  two  classifications: 

o  Problem  reporting  and  correction,  and 
o  Change  request  processing. 

Problem  reporting  and   correction   involves   a    formal    notification  Cusually 
consisting    of   a    form  which    is   filled    out   by  a  user)  of  an  error  in  the 
software.    The  error  is  then  verified  and  the  offending   moduleCs)  identified 
and    corrected.    The  effort  required  for  the  entire  process  is,  in  most  cases, 
minimal  in  that  any  more  than  a  very  minimal  redesign  is  seldom  necessary. 
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Change  request  processing  involves  a  formal  notification  (basically  consisting 
of  a  r®5uiraiients  statonent)  of  a  desired  enhancement,  alteration  or 
inprovanent  (e.g« 5  performance)  in  the  system's  functional  capabilities. 
Change    requests   always    require   some   software   redesign   and,  as  a  result, 

involve  a  greater  level  of  effort. 

Each  of  these  activities  are  illustrated  in  the  following  sections  with  some 
simple  examples  frcm  the  AIMS  software. 

TABLE  5.5-1 
EXAMPLE  4  SUMMARY 
SOFTWARE  MAINTENANCE 


SUBPHASES 

PROBLEM  REPORTING  AND  CORRECTION 

CHANGE  REQUEST  PROCESSING 

O 

System  Software 

o 

System  Software 

O 

System  Documentation 

o 

System  Documentation 

INPUT 

o 

System  Test  Cases 

o 

System  Test  Cases 

o 

Problem  Reports 

o 

Change  Requests 

OUTPUT 

O 

Updated  System  Software 

o 

Updated  System  Software 

o 

updated  System  Documentation 

o 

Updated  System  Documentation 

SUPPORTING 

o 

Fully  Automated  V,V&T  Tool  Set 

o 

Fully  Automated  V,V&T  Tool  Set 

TECHNOLOGY 

o 

Configuration  Control  Procedures 

o 

Configuration  Control  Procedures 

ACTIVITIES 

o 

Problem  Analysis 

o 

Requirements  Analysis 

o 

Problem  Correction 

o 

Redesign  and  Coding 

5.5.1  Problem  Reporting  and  Correction  Activity  Descriptions 

5.5.1.1*    Problem  Analysis 

When  a  problem  report  is  received,  the  problem  must  be  analyzed  to  determine 
the  cause  of  the  error.  Often,  the  error  needs  to  be  recreated  to  obtain 
additional  information  pertaining  to  the  source  of  the  error.  It  may  be  that 
a  user  has  made  an  error  which  was  the  result  of  a  misunderstanding  on  his 
part.  The  misunderstanding  could  be  the  user's  problem  or  it  may  be  the  user 
docunentation  that  is  incorrect. 

If  the  problem  is  indeed  an  error  in  the  software,  then  the  offending  code 
needs  to  be  identified  and  possible  solutions  addressed.  This  is  generally 
not  an  easy  task.  If  the  software  has  been  developed  using  the  lifecycle 
V,V&T  techniques  and  tools  described  in  the  previous  sections,  the  error  has 
escaped  very  thorou^  attempts  to  uncover  it.  Althou^  some  V,V&T  tools  can 
provide  assistance,  they  are  generally  not  very  helpful  in  locating  the  error. 
The  best  tool  to  use  is  one's  brain.  Tools  are  not  meant  to  replace  thinking 
but  to  assist  in  amplifying  the  thinking  process. 
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Once  the  error  has  been  located,  possible  solutions  can  be  identified.  These 
solutions  vary  from  being  very  trivial  code  modifications  to  an  extensive 
redesign.  It  is  important  to  note  that  if  good  lifecycle  V,V&T  practices  are 
utilized  during  software  development,  then  the  instances  of  redesign  are 
significantly  reduced. 

Example: 

An  AIMS  Problem  Report  is  received  describing  an  error  in  which  a  claims 
transaction  is  due  to  an  invalid  agent  nunber  althou^  that  particular  agent 
nunber  is  listed  in  the  user  docunentation  as  valid.  Upon  analysis  of  the 
problon,  the  cause  of  the  error  turns  out  to  be  that  a  new  agent  was  added 
to  the  company  with  a  new  number  assigned.  The  docunentation  was  updated 
but  the  table  of  agent  nunbers  in  the  AIMS  software  was  not  updated. 

5.5.1.2.    Problem  Correction 

When  the  source  of  an  error  is  found  to  be  in  the  software  and  the  offending 
code  is  identified,  the  correction  of  the  error  needs  to  be  made.  The  code  in 
the  module  which  contains  the  error  is  modified.  The  V,V&T  techniques  and 
tools  which  are  used  during  the  code  correction  process  will  depend  on  the 
extent  of  the  error.  In  the  previous  example,  only  a  table  addition  is 
required.  The  compiler  detects  most  errors  and  an  inspection  of  the  compiled 
table  is  generally  all  that  is  necessary  before  retesting.  In  a  case  where  an 
entire  algorithm  is  wrong,  a  formal  review  of  the  code  is  probably  necessary, 
as  well  as  use  of  the  full  automated  V,V&T  tool  set  to  completely  retest  the 
module. 

After  the  code  has  been  modified  and  module  testing  completed,  regression 
testing  is  performed.  In  cases  where  the  module  interfaces  with  another 
module(s),  interface  checking,  hierarchy,  module  calling,  etc.  tools  should 
be  employed  before  regression  checking.  All  system  conponents  which  can 
potentially  be  affected  by  the  change  are  retested  using  a  standard  set  of 
syst«n  test  cases  augmented  with  tests  aimed  at  revealing  the  original  error 
(e.g.,  the  actual  input  which  caused  the  error  should  be  included).  When 
testing  is  conplete,  the  actual  output  is  compared  with  the  expected  (correct) 
output,  possibly  using  a  file  comparator.  If  no  errors  are  present,  then  the 
updated  modules  are  incorporated  into  the  production  software  and  any 
pertinent  docunentation  is  updated. 

Example: 

An  error  was  discovered  where  on  midnight  of  December  30,  1980  the  year-to- 
date  premium  total  was  reset  so  that  premiums  received  on  December  31  were 
counted  in  the  1981  total  instead  of  the  1980  total.  Upon  examination,  it 
was  found  that  the  end  of  year  test  was  365  days.  1980,  however,  being  a 
leap  year,  contained  366  days  and  so  the  reset  occurred  a  day  early.  In 
this  case,  the  software  correction  is  quite  simple  and  is  implemented  with 
little  difficulty.  In  fact,  it  is  more  difficult  to  correct  the 
misinformation  in  the  database  than  it  is  to  correct  the  error. 
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5.5.2  Change  Request  Processing  Activity  Descriptions 

5.5.2.1.  Requirements  Analysis 

The  requirements  analysis  associated  with  system  enhancements  or  alterations 
is  essentially  the  same  as  described  earlier  in  this  chapter  except  for  one 
fundamental  difference.  For  all  practical  purposes,  the  feasibility  of  a 
particular  enhanced  or  altered  capability  depends  upon  how  well  that 
capability  can  be  implemented  within  the  existing  design.  The  more  redesign 
that  is  necessary,  the  greater  the  magnitude  of  the  cost  to  implement  it. 
Redesign  costs  are  not  an  issue  during  the  requirements  phase  of  the  initial 
software  developnent  since  there  is  no  existing  design.  What  this  means  is 
that  the  scope  of  what  is  or  is  not  feasible  is  much  narrower  during  the 
maintenance  phase  and  therefore  must  be  given  consideration. 

The  V,V&T  techniques  applied  during  this  requirements  analysis  are  the  same  as 
before  utilizing  reviews,  representation  aids,  etc.  to  assist  in  the  analysis 
and  specification. 

Example: 

High  level  canpary  management  issued  a  change  request  to  enhance  AIMS  to 
interactively  process  claims  and  premium  payment  transactions.  The  current 
system  batchs  the  transactions  on  a  file  which  is  then  processed  once  a  day. 
This  results  in  a  response  time  which  is  inconvenient  for  the  agents  as  well 
as  the  custaner.  It  is  decided  to  enhance  the  system  to  process  claims  and 
premium  payments  interactively  as  they  are  received. 

During  requirements  analysis,  it  is  found  that  those  system  modules  which 
are  impacted  by  the  redesign  are  those  which  control  the  inpuiy output  of  the 
batch  transactions.  Replacement  of  those  modules  with  ones  which  could 
perform  interactive  processing  is  relatively  straightforward.  Increased 
computer  usage  would,  however,  require  additional  hardware  resources  for 
normal  production  operation. 

Management  felt  that  the  extra  cost  was  worthwhile  and  thereby  authorized 
the  system  change. 

5.5.2.2.  Redesign  and  Coding 

The  process  followed  for  the  implementation  of  a  systan  enhancement  or  an 
alteration  is  no  different  than  that  followed  for  a  complete  software 
developnent  as  described  earlier  in  this  chapter.  Therefore,  no  further 
discussion  is  required. 
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CHAPTER  SIX 
SUMMARY 

This  document  has  presented  information  on  lifecycle  V,V&T  and  guidance  on 
planning  for  V,V&T.    It  has  covered: 

0  Software  development  phases  and  acccmparying  V,V&T  activities 

o  A  categorization  of  V,V&T  techniques  into  static,  dynamic,  and  formal 
analysis 

0  Suggestions  on  hew  to  integrate  static,  dynamic,  and  formal  V,V&T 
analysis  into  a  lifecycle 

0  Guidance  on  steps  and  approaches  to  take  when  planning  V ,V&T 
activities 

o  Recognition  that  V ,V&T  activities  must  be  tailored  to  meet  the  needs 
of  specific  projects  ; 

A  series  of  rather  detailed  examples  of  Y,V&T  approaches  using  increasing 
levels  of  automated  support  were  also  presented.  A  reference  guide  of 
techniques  and  tools  [POWE]  for  V,V&T  is  a  valuable  companion  for  this 
docunent.    A  glossary  of  V,V&T  terms  is  included  as  an  appendix. 
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GLOSSARY 

BLACK  BOX  TESTING:    see  FUNCTIONAL  TESTING 

BOUNDARY  VALUE  ANALYSIS:  a  selection  technique  in  which  test  data  is  chosen 
to  lie  along  "boundaries"  or  extrones  of  input  dcwiain  (or  output  range) 
classes,  data  structures,  procedure  parameters,  etc.  Choices  often  include 
maximum,  minimum,  and  trivial  values  or  parameters.  This  technique  is  often 
called  stress  testing. 

BRANCH  TESTING:  a  test  method  satisfying  coverage  criteria  that  require,  for 
each  decision  point,  each  possible  branch  be  executed  at  least  once. 

CAUSE  EFFECT  GRAPHING:  test  data  selection  technique.  The  inputs  and  outputs 
of  the  progran  are  determined  throu^  analysis  of  the  requirements.  A  minimal 
set  of  inputs  is  chosen  avoiding  the  testing  of  multiple  inputs  which  cause 
identical  output. 

COMPLETENESS:  the  property  that  all  necessary  parts  of  the  entity  in  question 
are  included.  Completeness  of  a  product  is  often  used  to  express  the  fact 
that  all  requirements  have  been  met  by  the  product. 

CONSISTENCY:  the  property  of  logical  coherency  amoung  constituent  parts. 
Consistency  may  also  be  expressed  as  adherence  to  a  given  set  of  rules. 

CORRECTNESS:  the  extent  to  which  software  is  free  from  design  and  coding 
defects,  i.e.  fault  free.  It  is  also  the  extent  to  which  software  meets  its 
specified  requirements  and  user  objectives.  (IEEE  Software  Engineering 
Terminology) 

DEBUGGING:  the  process  of  correcting  syntactic  and  logical  errors  detected 
during  coding.  With  the  primary  goal  of  obtaining  an  executing  piece  of  code, 
debugging  shares  with  testing  certain  techniques  and  strategies  but  differs  in 
its  usual  ad  hoc  application  and  local  scope. 

DESIGN  BASED  FUNCTIONAL  TESTING:  the  application  of  test  data  derived  through 
functional  analysis  (see  FUNCTIONAL  TESTING)  extended  to  include  design 
functions  as  well  as  requirement  functions. 

DRIVER:    code  which  sets  up  an  environment  and  calls  a  module  for  test. 

DYNAMIC  ANALYSIS:  involves  execution  or  simulation  of  a  development  phase 
product.  It  detects  errors  by  analyzing  the  response  of  a  product  to  sets  of 
input  data. 

EXTREMAL  TEST  DATA:  test  data  that  is  at  the  extremes,  or  boundaries,  of  the 
domain  of  an  input  variable  or  which  produces  results  at  the  boundaries  of  an 
output  domain. 

FORMAL  ANALYSIS:  uses  rigorous  mathematical  techniques  to  analyze  the 
algorithms  of  a  solution.  The  algorithms  may  be  analyzed  for  numerical 
properties,  efficiency,  and/or  correctness. 
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FUNCTIONAL  TESTING:  application  of  test  data  derived  from  the  specified 
functional  requirements  without  regard  to  the  final  program  structure, 

INSPECTION:    a  manual  analysis  technique  in  which  the   program  (requirements, 
design,    or   code)    is    examined    in   a    very  formal  and  disciplined  manner  to 
discover  errors. 

INSTRUMENTATION:  the  insertion  of  additional  code  into  the  program  in  order 
to  collect  information  about  program  behavior  during  program  execution. 

INVALID  INPUT  (TEST  DATA  FOR  INVALID  INPUT  DOMAIN):  test  data  that  lies 
outside  the  danain  of  the  program's  function. 

PATH  TESTING:  a  test  method  satisfying  coverage  criteria  that  each  logical 
path  through  the  program  be  tested.  Often  paths  throu^  the  program  are 
grouped  into  a  finite  set  of  classes;  one  path  from  each  class  is  then 
tested. 

PROOF  OF  CORRECTNESS:  the  use  of  techniques  of  mathematical  logic  to  infer 
that  a  relation  between  program  variables  assuned  true  at  program  entry 
implies  that  another  relation  between  program  variables  holds  at  program  exit. 

REGRESSION  TESTING: Rerunning  test  cases  which  a  program  has  previously 
executed  correctly  in  order  to  detect  errors  created  during  software 
correction  or  modification  activities. 

SIMULATION:  use  of  an  executable  model  to  represent  the  behavior  of  an 
object.  During  testing  the  computational  hardware,  the  external  environment, 
and  even  code  segments  may  be  simulated. 

SPECIAL  TEST  DATA:  test  data  based  on  input  values  that  are  likely  to  require 
special  handling  by  the  program. 

STATEMENT  TESTING:  a  test  method  satisfying  the  criterion  that  each  statement 
in  a  program  be  executed  at  least  once  during  program  testing. 

STATIC  ANALYSIS:  direct  analysis  of  the  form  and  structure  of  a  product 
without  executing  the  product.  It  may  be  applied  to  the  requirements,  design 
or  code. 

STRESS  TESTING:    see  BOUNDARY  VALUE  ANALYSIS. 

STUB:  special  code  segments  that  when  invoked  by  a  code  segment  under  test 
will  simulate  the  behavior  of  designed  and  specified  modules  not  yet 
constructed. 

SYNBOLIC  EXECUTION:  an  analysis  technique  that  derives  a  symbolic  expression 
for  each  progran  path. 

TEST  DATA  SET  :    set  of  input  elements  used  in  the  testing  process. 
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TEST  DRIVER:  a  program  which  directs  the  execution  of  another  program  against 
a  collection  of  test  data  sets.  Usually,  the  test  driver  records  and 
organizes  the  output  generated  as  the  tests  are  run. 

TEST  HARNESS:    see  TEST  DRIVER. 

TESTING:  examination  of  the  behavior  of  a  program  by  executing  the  program  on 
sample  data  sets. 

VALID  INPUT  (TEST  DATA  FOR  A  VALID  INPUT  DOMAIN):  test  data  that  lies  within 
the  dcanain  of  the  function  represented  by  the  program. 

VALIDATION:  determination  of  the  correctness  of  the  final  program  or  software 
produced  from  a  development  project  with  respect  to  the  user  needs  and 
requirements. 

VERIFICATION:  in  general,  the  demonstration  of  consistency,  canpleteness,  and 
correctness  of  the  software  at  each  stage  and  between  each  stage  of  the 
development  lifecycle, 

WALKTHRCXJGH:  a  manual  analysis  technique  in  which  the  module  author  describes 
the  module's  structure  and  logic  to  an  audience  of  colleagues. 


NOTE:    Most  of  the  definitions  above  are  from: 

ADRION,W.R.  ,BRANSTAD,M.A.  ,and  CHERNIAVSKY,  J.C. , "Validation,  Verification, 
and  Testing,"  NBS  Special  Publicaiton  500-75. 


/ 
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